SHF: Small: A General Framework for Accelerating AI on Resource-Constrained Edge Devices

边缘设备普遍使用的上升趋势为未来移动的和物联网应用中的设备智能提供了绝佳的机会，包括移动的增强现实（AR）/虚拟现实（VR）、智能制造、移动的医疗保健和自动驾驶汽车。虽然这些边缘设备具有完整的软件/硬件堆栈来执行机器学习模型，但它们通常具有有限的计算资源。他们无法直接执行机器学习模型。为了跟上快速增长的移动的和物联网应用的部署，迫切需要设计新的神经网络架构，以加速资源受限的边缘设备上的人工智能（AI）。该提案旨在开发一种新的框架，可以有效地设计适合在边缘设备上执行的神经网络架构。该框架开发了同时平衡内存成本、计算效率和预测准确性的网络架构，可以推进具有低延迟和高效率要求的设备上AI应用。新的部署优化方法通常可以使神经网络在异构商品计算平台上的实现和部署受益，而无需定制硬件。该项目将为与计算架构设计和边缘计算系统相关的广泛研究课题奠定坚实的基础。研究成果将为跨学科的大学生/研究生和少数民族学生提供新的研究课题和任务。本项目开发了一个整体框架，用于设计考虑边缘设备硬件限制的高效和有效的神经网络架构。它首先开发了一种自动化的硬件感知神经网络架构设计方法，以有效地生成最佳的神经网络架构，可以平衡所需的精度和计算性能之间的权衡。进一步的调查进行开发新的神经网络优化方法，以减少内存占用和计算成本在细粒度的方式，同时满足精度要求。设计了新的剪枝方法，准确跟踪网络参数的重要性，有效减少剪枝迭代次数和浮点运算。此外，开发了新的实现机制，如权重共享感知微调，动态分区和按需加载方案，以最大限度地减少加载时间开销，并使设计的架构在边缘设备上的有效部署和评估。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The upward trend of the pervasive usage of edge devices provides excellent opportunities for on-device intelligence in future mobile and IoT applications, including mobile augmented reality (AR)/Virtual reality (VR), smart manufacturing, mobile healthcare, and autonomous vehicles. While these edge devices have complete software/hardware stacks to execute machine-learning models, they usually have constrained computing resources. They cannot afford to execute the machine-learning models directly. To keep up with the fast-growing deployment of mobile and IoT applications, it is urgently needed to design new neural-network architectures for accelerating artificial intelligence (AI) on resource-constrained edge devices. This proposal aims to develop a novel framework that can efficiently design neural-network architectures suitable for execution on edge devices. The proposed framework develops network architectures that simultaneously balance memory cost, computing efficiency, and prediction accuracy, which can advance on-device AI applications with low-latency and high-efficiency requirements. The new deployment optimization methods can generally benefit neural-network implementation and deployment on heterogeneous commodity computing platforms without customized hardware. The project will lead to a solid foundation for a broad range of research topics related to computing architecture design and edge-computing systems. The research results can benefit interdisciplinary curriculums with new research topics and tasks for undergraduate/graduate and minority students.This project develops a holistic framework for designing efficient and effective neural-network architectures considering edge devices’ hardware constraints. It first develops an automated hardware-aware neural-network architecture-design method to efficiently generate optimal neural-network architectures that can balance the trade-offs between the required accuracy and computational performance. A further investigation is conducted to develop novel neural-network optimization methods to reduce memory footprints and computational costs in a fine-grained way while satisfying the accuracy requirement. New pruning methods are designed to accurately track the importance of network parameters and effectively reduce pruning iterations and floating-point operations. Moreover, novel implementation mechanisms, such as weight-sharing-aware fine-tuning, dynamic partitioning, and on-demand loading schemes, are developed to minimize the loading time overhead and enable efficient deployment and evaluation of the designed architectures on edge devices. Commodity edge devices and FPGAs are employed to implement and evaluate the designed neural-network architecture.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.