SHF: Medium: Training Sparse Neural Networks with Co-Designed Hardware Accelerators: Enabling Model Optimization and Scientific Exploration

机器学习系统是近乎完美的自动语音识别、自动驾驶汽车、计算机视觉和自然语言理解等新技术的关键驱动力。其中许多系统的底层推理引擎都是基于神经网络的。在神经网络可以用于这些推理任务之前，它必须使用已知输入输出对的数据语料库进行训练。此培训过程计算非常密集，当前的系统需要在图形处理单元(GPU)或云中的中央处理单元上花费数周至数月的时间。随着更多的数据可用，训练时间长的问题进一步加剧，因为需要更大、更有效的网络模型。对神经网络的理论理解是有限的，因此实验和经验优化仍然是理解深层神经网络和在该领域创新的主要工具。然而，进行更大规模实验的能力正集中在少数拥有必要财政和计算资源的大型实体身上。即使对于那些拥有这种资源的人来说，训练神经网络的痛苦漫长的实验周期意味着不会对神经网络模型结构进行大规模搜索和优化。该研究项目的最终目标是通过使用硬件加速器，实现大规模神经网络训练和模型优化的大众化和分散化。将培训时间从几周减少到几个小时，将允许研究人员进行更多的实验，获得深入学习系统的基本内部工作原理的知识。与现有的基于GPU的训练模式相比，硬件加速器的能效也更高，因此该项目的进步可以显著降低神经网络训练任务所需的能源消耗。该项目包括一个跨越理论、硬件架构和设计、软件控制和系统集成的跨学科研究计划。一类新的具有预定义稀疏性的神经网络正在被探索。这些稀疏神经网络与非常灵活、高速、高能效的硬件架构共同设计，可在给定的现场可编程门阵列(FPGA)芯片中最大限度地提高任何型号的电路速度。这种算法-硬件协同设计是一个关键的研究主题，它区别于以前的研究，以前的研究在训练过程中以与并行硬件加速不兼容的方式实施一些稀疏性。特别是，提出的体系结构同时在每个网络层上运行，并行执行前向和后向传播，并完全跨层流水线执行。在高精度运算的情况下，预计相对于GPU的速度将提高约5倍。使用对数域算法，这些增益预计将增加到100倍或更大。目前正在开发软件和算法来管理多块FPGA板，从而简化和自动化模型搜索和培训过程。这些算法利用重新配置现场可编程门阵列的能力，以速度换取精度，这是GPU所缺乏的能力。这些软件工具还将成为通向机器学习社区使用的流行Python库的桥梁。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Machine learning systems are critical drivers of new technologies such as near-perfect automatic speech recognition, autonomous vehicles, computer vision, and natural language understanding. The underlying inference engine for many of these systems is based on neural networks. Before a neural network can be used for these inference tasks, it must be trained using a data corpus of known input-output pairs. This training process is very computationally intensive with current systems requiring weeks to months of time on graphic processing units (GPUs) or central processing units in the cloud. As more data becomes available, this problem of long training time is further exacerbated because larger, more effective network models become desirable. The theoretical understanding of neural networks is limited, so experimentation and empirical optimization remains the primary tool for understanding deep neural networks and innovating in the field. However, the ability to conduct larger scale experiments is becoming concentrated with a few large entities with the necessary financial and computational resources. Even for those with such resources, the painfully long experimental cycle for training neural networks means that large-scale searches and optimizations over the neural network model structure are not performed. The ultimate goal of this research project is to democratize and distribute the ability to conduct large scale neural network training and model optimizations at high speed, using hardware accelerators. Reducing the training time from weeks to hours will allow researchers to run many more experiments, gaining knowledge into the fundamental inner workings of deep learning systems. The hardware accelerators are also much more energy efficient than the existing GPU-based training paradigm, so advances made in this project can significantly reduce the energy consumption required for neural network training tasks.This project comprises an interdisciplinary research plan that spans theory, hardware architecture and design, software control, and system integration. A new class of neural networks that have pre-defined sparsity is being explored. These sparse neural networks are co-designed with a very flexible, high-speed, energy-efficient hardware architecture that maximizes circuit speed for any model size in a given Field Programmable Gate Array (FPGA) chip. This algorithm-hardware co-design is a key research theme that differentiates this approach from previous research that enforces some sparsity during the training process in a manner incompatible with parallel hardware acceleration. In particular, the proposed architecture operates on each network layer simultaneously, executing the forward- and back-propagation in parallel and pipelined fully across layers. With high precision arithmetic, a speed-up of about 5X relative to GPUs is expected. Using log-domain arithmetic, these gains are expected to increase to 100X or larger. Software and algorithms are being developed to manage multiple FPGA boards, simplifying and automating the model search and training process. These algorithms exploit the ability to reconfigure the FPGAs to trade speed for accuracy, a capability lacking in GPUs. These software tools will also serve as a bridge to popular Python libraries used by the machine learning community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.