SHF: Small: A General Framework for Accelerating AI on Resource-Constrained Edge Devices

边缘设备普遍使用的上升趋势为未来移动和物联网应用中的设备智能提供了极好的机会，包括移动增强现实(AR)/虚拟现实(VR)、智能制造、移动医疗保健和自动驾驶汽车。虽然这些边缘设备拥有完整的软件/硬件堆栈来执行机器学习模型，但它们通常具有有限的计算资源。他们承担不起直接执行机器学习模型的代价。为了跟上快速增长的移动和物联网应用部署，迫切需要设计新的神经网络体系结构，以在资源受限的边缘设备上加速人工智能(AI)。该方案旨在开发一种新的框架，能够有效地设计适合在边缘设备上执行的神经网络结构。该框架开发了同时平衡存储成本、计算效率和预测精度的网络体系结构，可以促进具有低延迟和高效率要求的设备上AI应用。新的部署优化方法一般可以在不需要定制硬件的情况下，在不同的商用计算平台上实现和部署神经网络。该项目将为与计算架构设计和边缘计算系统相关的广泛研究课题奠定坚实的基础。这项研究成果可以为跨学科课程的本科生/研究生和少数族裔学生提供新的研究主题和任务。该项目开发了一个整体框架，用于设计考虑边缘设备硬件限制的高效和有效的神经网络结构。它首先开发了一种自动化的硬件感知神经网络结构设计方法，以有效地生成能够在所需精度和计算性能之间权衡的最佳神经网络结构。进一步研究开发新的神经网络优化方法，在满足精度要求的同时，以细粒度的方式减少存储空间和计算成本。设计了新的剪枝方法，准确跟踪网络参数的重要性，有效减少剪枝迭代和浮点运算。此外，还开发了新的实现机制，如加权感知微调、动态分区和按需加载方案，以最大限度地减少加载时间开销，并使所设计的体系结构能够在边缘设备上高效部署和评估。使用商用边缘设备和现场可编程门阵列来实施和评估设计的神经网络结构。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The upward trend of the pervasive usage of edge devices provides excellent opportunities for on-device intelligence in future mobile and IoT applications, including mobile augmented reality (AR)/Virtual reality (VR), smart manufacturing, mobile healthcare, and autonomous vehicles. While these edge devices have complete software/hardware stacks to execute machine-learning models, they usually have constrained computing resources. They cannot afford to execute the machine-learning models directly. To keep up with the fast-growing deployment of mobile and IoT applications, it is urgently needed to design new neural-network architectures for accelerating artificial intelligence (AI) on resource-constrained edge devices. This proposal aims to develop a novel framework that can efficiently design neural-network architectures suitable for execution on edge devices. The proposed framework develops network architectures that simultaneously balance memory cost, computing efficiency, and prediction accuracy, which can advance on-device AI applications with low-latency and high-efficiency requirements. The new deployment optimization methods can generally benefit neural-network implementation and deployment on heterogeneous commodity computing platforms without customized hardware. The project will lead to a solid foundation for a broad range of research topics related to computing architecture design and edge-computing systems. The research results can benefit interdisciplinary curriculums with new research topics and tasks for undergraduate/graduate and minority students.This project develops a holistic framework for designing efficient and effective neural-network architectures considering edge devices’ hardware constraints. It first develops an automated hardware-aware neural-network architecture-design method to efficiently generate optimal neural-network architectures that can balance the trade-offs between the required accuracy and computational performance. A further investigation is conducted to develop novel neural-network optimization methods to reduce memory footprints and computational costs in a fine-grained way while satisfying the accuracy requirement. New pruning methods are designed to accurately track the importance of network parameters and effectively reduce pruning iterations and floating-point operations. Moreover, novel implementation mechanisms, such as weight-sharing-aware fine-tuning, dynamic partitioning, and on-demand loading schemes, are developed to minimize the loading time overhead and enable efficient deployment and evaluation of the designed architectures on edge devices. Commodity edge devices and FPGAs are employed to implement and evaluate the designed neural-network architecture.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.