SHF: Small: NPU-based Architecture for Accelerating Deep Learning on Mobile Devices

深度学习技术的快速进步使得许多新兴的人工智能应用（例如增强现实）成为可能，并且在移动设备上运行这些应用程序的需求巨大。然而，深度学习模型本质上是计算密集型的，这使得它们在电池供电的移动设备上部署具有挑战性。该项目通过设计基于神经处理单元（NPU）的移动架构，系统地研究了在移动设备上运行深度学习应用程序的基本和具有挑战性的问题。 NPU 是专门用于加速深度学习算法的微处理器；然而，它会带来准确性损失，解决这个问题是一个挑战。这项研究确定了在 NPU 上运行深度学习模型的一些特殊特征，并利用这些发现来设计新颖的技术，以根据应用要求最大限度地提高准确性或最大限度地缩短处理时间。由于深度学习已成功应用于人们日常生活中的各种问题，该项目通过提高移动设备上运行深度学习应用程序的性能、能源效率和质量，具有巨大的造福社会的潜力。该项目还通过开发新课程、传播教育和培训研究、让代表性不足的学生参与研究以及向高中生进行推广等方式为社会做出贡献。该项目的主要目标是设计一种基于 NPU 的架构来加速深度学习，以解决 NPU 的准确性损失问题以及当前移动架构的能量和性能限制。该项目包括三个任务：（1）研究模型分区技术，将深度学习模型分解为在异构处理器上运行的不同层，以根据应用程序需求最小化处理时间或最大化准确性； (2) 设计能源和热感知架构，通过探索分解异构处理器之间的计算以避免过热的技术，解决当前移动架构的性能限制； (3) 根据在 NPU 上执行深度学习模型的置信水平，研究如何以及在何处运行计算，探索边缘/服务器、硬件加速器和基于 NPU 的架构之间的协作智能，以优化性能。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

The rapid progress of deep-learning techniques has enabled many emerging artificial intelligence applications (e.g., augmented reality), and there is a tremendous demand for running these applications on mobile devices. However, deep-learning models are by nature computationally intensive, making them challenging to deploy on battery-powered mobile devices. This project systematically investigates the fundamental and challenging issues for running deep-learning applications on mobile devices by designing a mobile architecture based on Neural Processing Units (NPUs). An NPU is a microprocessor that specializes in the acceleration of deep-learning algorithms; however, it incurs accuracy loss, and it is a challenge to address this problem. This research identifies some special characteristics of running deep-learning models on NPUs and leverages such findings to design novel techniques to maximize accuracy or minimize processing time based on the application requirements. As deep learning has been successfully applied to various problems in people's daily lives, this project has great potential to benefit society by improving the performance, the energy efficiency, and the quality of running deep-learning applications on mobile devices. This project is also contributing to society through developing new curricula, disseminating research for education and training, engaging under-represented students in research, and outreaching to high-school students.The primary goal of this project is to design an NPU-based architecture for accelerating deep learning that can address the accuracy-loss problem of NPUs as well as the energy and performance limitations of current mobile architectures. The project consists of three tasks: (1) investigating model-partitioning techniques to decompose the deep-learning model into different layers running on heterogeneous processors to minimize processing time or maximize accuracy based on the application requirements; (2) designing energy-and thermal-aware architectures to address the performance limitations of the current mobile architecture, by exploring techniques to decompose the computation between heterogeneous processors to avoid overheating; (3) exploring the collaborative intelligence among edge/servers, hardware accelerators, and NPU-based architectures to optimize performance, by investigating how and where to run the computation based on the confidence level of executing deep learning models on an NPU.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.