Neural network topologies from a hardware perspective

当前工业革命的核心在于机器学习算法的发展：尤其是深度神经网络（dnn）。深度神经网络被广泛用于大量的人工智能（AI）任务，包括计算机视觉、语音识别和自然语言处理。我努力解决的问题是，最先进的深度神经网络以牺牲巨大的计算复杂度为代价来达到高精度。机器学习研究人员的目标是设计架构来提高准确性，而不关注推理成本。我渴望从机器学习和硬件设计自动化的角度研究深度神经网络架构。我的目标是弥合神经网络领域和定制硬件领域研究之间的差距，推动精度和计算成本之间权衡的极限。Wang等人分析了dnn的算法进步如何有利于专用硬件，并展示了与其他平台相比，现场可编程门阵列（fpga）如何达到卓越的性能。在考虑深度学习算法的设计时，我被驱使去研究FPGA平台的能力和可用资源，并对DNN拓扑设计有更深入的了解，以便充分利用硬件。底层FPGA架构由k输入布尔查找表（lut）组成。为了利用这些lut提供的灵活性，E. Wang等人引入了LUTNet，这是第一个以k输入lut作为推理算子的神经网络架构。研究表明，利用lut的潜力及其能力可能导致更重的网络修剪，从而显著减少面积，同时保持精度[3]。二进制神经网络（bnn）解决了神经网络计算成本高和在内存受限平台上部署不可行性的问题。G. Constantinides教授解释了为什么与其他数据表示的网络相比，bnn并不普遍能够达到如此高的准确率。原因不在于数据表示的通用性，而在于传统的设计技术无法使网络拓扑适应底层数据类型。因此，向前推进，我们需要找到一种方法来适应神经网络拓扑结构的数据和离散表示激活的性质。因此，通过将“布尔电路视为神经网络”来确定有价值的拓扑研究的可能性。我最初想找到答案的问题是：什么样的网络拓扑最适合LUTNet，对LUTNet或拓扑本身进行什么样的进一步修改将提高效率？为了回答这个问题，我会尝试调查硬件中出现的问题，例如，长关键路径，高风扇输入/输出或内存限制，并找到解决这些问题的现有/开发的新拓扑。接下来，我将探讨哪些修改将有利于设计，这些修改来自机器学习和硬件的角度。

At the core of the current industrial revolution lies the growth of machine learning algorithms: especially Deep Neural Networks (DNNs) [1]. DNNs are widely used for a vast number of artificial intelligence (AI) tasks, including computer vision, speech recognition, and natural language processing. The problem I strive to address is that state-of-the-art DNNs reach high accuracy at the expense of large computational complexity [1]. Machine learning researchers aim to design architectures to improve accuracy, without focusing on the inference cost [1]. I aspire to investigate DNN architectures from both machine learning and hardware design automation perspectives. My goal is to bridge the gap between the research in the neural network field and custom hardware field, to push the limits of the trade-off between accuracy and computational cost.E. Wang et al. [2] analysed how DNNs' algorithmic advancement favours specialised hardware and showed how Field Programmable Gate Arrays (FPGAs) reach superior performance in comparison to alternative platforms. I am driven to study the capabilities of and resources available in FPGA platforms and gain deeper understanding of DNN topological design to take full advantage of the hardware when thinking about the design of the algorithms for Deep Learning.The underlying FPGA architecture consists of K-input Boolean Lookup Tables (LUTs). To take advantage of the flexibility provided by these LUTs, E. Wang et al. [3] introduced LUTNet, the first neural network architecture featuring K-input LUTs as inference operators. It was shown that taking advantage of the potential of the LUTs and their capabilities results in the possibility of heavier network pruning, and thus significant area reduction, while maintaining accuracy [3].Binary Neural Networks (BNNs) address the high computational cost of neural networks and the infeasibility of deployment on memory-constrained platforms. Prof. G. Constantinides [4] explains why BNNs are not universally able to reach as high accuracy compared with networks with other data representations. The reason does not lie within the generality of the data representation, but rather in the traditional design technique that is unable to adapt the topology of the network to the underlying datatype. Hence, moving forward, we need to find a way to adapt the neural network topology to both the data and the nature of the discrete representation of activations. Therefore, the possibility of valuable topological research is identified by considering "Boolean circuits as neural networks".The question I would initially like to find the answer for is: What network topologies are best suited for LUTNet and what further modifications to LUTNet or the topology itself would lead to further efficiency? To answer the question, I would try to investigate what are the problems that come up in hardware, for example, long critical paths, high fan-in/-out or memory constraints and find existing/developing new topologies that address these problems. Following, I would explore what modifications would benefit the design, these modifications coming from both machine learning and hardware perspectives.