Collaborative Research: SHF: Medium: TensorNN: An Algorithm and Hardware Co-design Framework for On-device Deep Neural Network Learning using Low-rank Tensors

深度神经网络(DNN)是一种重要的人工智能(AI)技术，近年来在图像识别、机器翻译、自动驾驶汽车和医疗诊断等领域获得了广泛的应用。传统的DNN是使用云计算实现的，在云计算中，大量计算资源以集中池化的方式可用。为了实现更强的数据保密性、更短的响应时间和更轻松的数据传输负担，在网络边缘以分布式方式部署DNN功能已经成为一个非常有吸引力的命题。然而，在处于网络边缘的移动设备上进行DNN学习是非常具有挑战性的，因为大量时间和能源消耗的要求相互冲突，而设备上的资源有限。为了应对这一挑战，该项目利用低阶张量作为表示和压缩张量格式数据的强大数学工具，形成了一系列新的超低成本深度神经网络。这为深度神经网络学习带来了数量级的时间和能量消耗。在BigData研究的许多领域进行的调查也将受益。这个项目包括研究生和本科生，特别是来自代表性不足的群体，通过暑期研究经验和高级设计项目来扩大对计算的参与。该项目的成果将以学术机构和产业界的技术出版物、讲座和教程的形式向社会传播。为了消除在资源和能量受限的嵌入式设备上实现实时节能DNN学习的障碍，本项目在三个层面上考虑了创新：1)在理论层面，提出了一种新的无冗余矩阵向量乘法方案，以降低计算成本，包括一种新的低阶张量在线更新方案，以实现快速压缩数据更新；2)在算法层面，提出了支持低成本加速推理和训练的基于低阶张量的正向和反向传播方案，包括灾难性遗忘恢复训练方案和训练感知压缩方案，提高了学习的稳健性和存储效率；3)在硬件设计层面，提出了高效的硬件结构，充分利用了低阶张量的优点，提高了设备上DNN推理和学习的硬件性能。最后，将通过在不同目标应用的不同DNN模型上的软件实现来验证和评估所提出的研究的有效性。还将开发基于现场可编程门阵列(FPGA)的硬件原型。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Deep neural network (DNN) is an important Artificial Intelligence (AI) technique and it has recently gained widespread applications in numerous fields such as image recognition, machine translation, autonomous vehicles and healthcare diagnosis. Conventional DNNs are implemented using cloud computing, where a large amount of computing resource is available in a centrally-pooled manner. In order to achieve stronger data privacy, less response time and relaxed data transmission burden, deploying DNN functionality in a distributed manner at the edges of the network has become a very attractive proposition. However, DNN-learning on mobile devices that are at the edge of the network is very challenging due to conflicting requirements of large time and energy consumption, and limited on-device resources. In order to address this challenge, this project leverages low-rank tensors as a powerful mathematical tool for representing and compressing tensor-format data, to form a new family of ultra-low cost deep neural networks. This brings an order-of-magnitude reduction in time and energy consumption for deep neural network learning. Investigations in many areas of BigData research will benefit as well. This project involves graduate and undergraduate students, especially from underrepresented groups, through summer research experiences, and senior design projects to broaden the participation of computing. The outcomes of this project will be disseminated to the community in the format of technical publications, talks and tutorials in both academic institutions and industry.In order to remove the barriers of realizing real-time energy-efficient DNN-learning on the resource and energy-constrained embedded devices, this project considers innovations at three levels: 1) at theory level, it develops a novel redundancy-free matrix-vector multiplication scheme to reduce computational cost, including a new online update scheme for low-rank tensors to enable fast compressed data update; 2) at algorithm level, it develops low-rank tensor-based forward and backward propagation schemes to support low-cost accelerated inference and training, including catastrophic forgetting-resilient training scheme and training-aware compression scheme to improve the learning robustness and memory efficiency; and 3) at hardware design level, it proposes efficient hardware architecture that fully utilize the benefits provided by low-rank tensors to achieve improved hardware performance for on-device DNN inference and learning. Finally, the efficacy of the proposed research will be validated and evaluated, via software implementations on different DNN models in different target applications. A field-programmable gate array (FPGA)-based hardware prototype will also be developed.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.