Neural network topologies from a hardware perspective

当前工业革命的核心是机器学习算法的增长：特别是深度神经网络（DNN）[1]。DNN广泛用于大量人工智能（AI）任务，包括计算机视觉、语音识别和自然语言处理。我努力解决的问题是，最先进的DNN以巨大的计算复杂度为代价达到高精度[1]。机器学习研究人员旨在设计架构以提高准确性，而不关注推理成本[1]。我渴望从机器学习和硬件设计自动化的角度研究DNN架构。我的目标是弥合神经网络领域和定制硬件领域的研究之间的差距，推动精度和计算成本之间的权衡。Wang等人[2]分析了DNN的算法进步如何有利于专用硬件，并展示了现场可编程门阵列（FPGA）如何与其他平台相比达到上级性能。我被驱使研究FPGA平台的功能和可用资源，并深入了解DNN拓扑设计，以便在考虑深度学习算法的设计时充分利用硬件。底层FPGA架构由K输入布尔运算表（LUT）组成。为了利用这些LUT提供的灵活性，E. Wang等人[3]介绍了LUTNet，这是第一个以K输入LUT作为推理算子的神经网络架构。研究表明，利用LUT的潜力及其功能可以实现更重的网络修剪，从而显著减少面积，同时保持准确性[3]。二进制神经网络（BNN）解决了神经网络的高计算成本和在内存受限平台上部署的不可行性。G.教授Constantinides [4]解释了为什么与其他数据表示的网络相比，BNN不能普遍达到同样高的准确性。原因不在于数据表示的一般性，而在于传统的设计技术无法使网络拓扑适应底层数据类型。因此，向前迈进，我们需要找到一种方法，使神经网络拓扑结构适应数据和激活的离散表示的性质。因此，有价值的拓扑研究的可能性是通过考虑“布尔电路作为神经网络”来确定的。我最初想找到答案的问题是：什么样的网络拓扑结构最适合LUTNet，以及对LUTNet或拓扑结构本身的进一步修改将导致进一步的效率？为了回答这个问题，我会尝试调查硬件中出现的问题，例如，长关键路径，高扇入/扇出或内存限制，并找到解决这些问题的现有/开发新拓扑。接下来，我将探讨哪些修改会使设计受益，这些修改来自机器学习和硬件的角度。

At the core of the current industrial revolution lies the growth of machine learning algorithms: especially Deep Neural Networks (DNNs) [1]. DNNs are widely used for a vast number of artificial intelligence (AI) tasks, including computer vision, speech recognition, and natural language processing. The problem I strive to address is that state-of-the-art DNNs reach high accuracy at the expense of large computational complexity [1]. Machine learning researchers aim to design architectures to improve accuracy, without focusing on the inference cost [1]. I aspire to investigate DNN architectures from both machine learning and hardware design automation perspectives. My goal is to bridge the gap between the research in the neural network field and custom hardware field, to push the limits of the trade-off between accuracy and computational cost.E. Wang et al. [2] analysed how DNNs' algorithmic advancement favours specialised hardware and showed how Field Programmable Gate Arrays (FPGAs) reach superior performance in comparison to alternative platforms. I am driven to study the capabilities of and resources available in FPGA platforms and gain deeper understanding of DNN topological design to take full advantage of the hardware when thinking about the design of the algorithms for Deep Learning.The underlying FPGA architecture consists of K-input Boolean Lookup Tables (LUTs). To take advantage of the flexibility provided by these LUTs, E. Wang et al. [3] introduced LUTNet, the first neural network architecture featuring K-input LUTs as inference operators. It was shown that taking advantage of the potential of the LUTs and their capabilities results in the possibility of heavier network pruning, and thus significant area reduction, while maintaining accuracy [3].Binary Neural Networks (BNNs) address the high computational cost of neural networks and the infeasibility of deployment on memory-constrained platforms. Prof. G. Constantinides [4] explains why BNNs are not universally able to reach as high accuracy compared with networks with other data representations. The reason does not lie within the generality of the data representation, but rather in the traditional design technique that is unable to adapt the topology of the network to the underlying datatype. Hence, moving forward, we need to find a way to adapt the neural network topology to both the data and the nature of the discrete representation of activations. Therefore, the possibility of valuable topological research is identified by considering "Boolean circuits as neural networks".The question I would initially like to find the answer for is: What network topologies are best suited for LUTNet and what further modifications to LUTNet or the topology itself would lead to further efficiency? To answer the question, I would try to investigate what are the problems that come up in hardware, for example, long critical paths, high fan-in/-out or memory constraints and find existing/developing new topologies that address these problems. Following, I would explore what modifications would benefit the design, these modifications coming from both machine learning and hardware perspectives.