Neural network topologies from a hardware perspective

点击翻译按钮获取中文摘要

At the core of the current industrial revolution lies the growth of machine learning algorithms: especially Deep Neural Networks (DNNs) [1]. DNNs are widely used for a vast number of artificial intelligence (AI) tasks, including computer vision, speech recognition, and natural language processing. The problem I strive to address is that state-of-the-art DNNs reach high accuracy at the expense of large computational complexity [1]. Machine learning researchers aim to design architectures to improve accuracy, without focusing on the inference cost [1]. I aspire to investigate DNN architectures from both machine learning and hardware design automation perspectives. My goal is to bridge the gap between the research in the neural network field and custom hardware field, to push the limits of the trade-off between accuracy and computational cost.E. Wang et al. [2] analysed how DNNs' algorithmic advancement favours specialised hardware and showed how Field Programmable Gate Arrays (FPGAs) reach superior performance in comparison to alternative platforms. I am driven to study the capabilities of and resources available in FPGA platforms and gain deeper understanding of DNN topological design to take full advantage of the hardware when thinking about the design of the algorithms for Deep Learning.The underlying FPGA architecture consists of K-input Boolean Lookup Tables (LUTs). To take advantage of the flexibility provided by these LUTs, E. Wang et al. [3] introduced LUTNet, the first neural network architecture featuring K-input LUTs as inference operators. It was shown that taking advantage of the potential of the LUTs and their capabilities results in the possibility of heavier network pruning, and thus significant area reduction, while maintaining accuracy [3].Binary Neural Networks (BNNs) address the high computational cost of neural networks and the infeasibility of deployment on memory-constrained platforms. Prof. G. Constantinides [4] explains why BNNs are not universally able to reach as high accuracy compared with networks with other data representations. The reason does not lie within the generality of the data representation, but rather in the traditional design technique that is unable to adapt the topology of the network to the underlying datatype. Hence, moving forward, we need to find a way to adapt the neural network topology to both the data and the nature of the discrete representation of activations. Therefore, the possibility of valuable topological research is identified by considering "Boolean circuits as neural networks".The question I would initially like to find the answer for is: What network topologies are best suited for LUTNet and what further modifications to LUTNet or the topology itself would lead to further efficiency? To answer the question, I would try to investigate what are the problems that come up in hardware, for example, long critical paths, high fan-in/-out or memory constraints and find existing/developing new topologies that address these problems. Following, I would explore what modifications would benefit the design, these modifications coming from both machine learning and hardware perspectives.