Collaborative Research: Enabling Intelligent Cameras in Internet-of-Things via a Holistic Platform, Algorithm, and Hardware Co-design

将深度神经网络（DNN）驱动的功能引入物联网（IoT）设备以实现无处不在的智能“物联网摄像头”的巨大需求。然而，最先进的深度神经网络具有令人望而却步的能源成本，使得它们在资源受限的物联网平台中部署不切实际。该项目将通过平台、硬件和算法协同设计创新的系统集成，开发一种新型节能深度神经网络框架。尽管人们对节能深度神经网络的兴趣日益浓厚，但现有技术缺乏从系统到算法再到硬件实现的整个设计抽象堆栈的系统优化。拟议的研究主张通过共同优化平台、硬件和算法级的协同设计工作，对节能和自适应dnn驱动的“物联网摄像头”进行创新、全面的努力。在系统层面，我们将解决如何自动生成和适应DNN模型和实现，以满足各种“物联网设备”特定应用的性能需求和特定设备的资源约束。在硬件层面，我们将利用DNN激活中观察到的高稀疏性，通过使用低成本的零预测器来实现DNN训练和推理的节能硬件实现，从而绕过不必要的计算。在算法层面，我们将在深度神经网络训练中开发创新的因式稀疏正则化，以及高效、可控的自适应推理机制，与我们的硬件创新充分互补并紧密结合。所提出的研究将推进每个层次的科学领域，从系统和算法，硬件和一个整体的，系统的跨层次的方法来设计节能智能系统。该项目的进展将使无处不在的dnn驱动的智能功能在资源有限的日常生活设备中得到显著增加，包括许多基于摄像头的物联网（IoT）应用，如交通监控、自动驾驶和智能汽车、个人数字助理、监控和安全以及增强现实。随着基于摄像头的物联网设备渗透到各行各业，通过使dnn驱动的智能在这些设备中无处不在，拟议的研究可以对全球社会和经济产生巨大影响。该研究将与节能深度学习教育相结合。教育活动包括课程开发、本科研究和向K-12学生推广。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

There has been a tremendous demand for bringing Deep Neural Network (DNN) powered functionality into Internet of Thing (IoT) devices to enable ubiquitous intelligent "IoT cameras". However, state-of-the-art DNNs have a prohibitive energy cost, making them impractical to be deployed in resource-constrained IoT platforms. This project will develop a novel energy-efficient DNN framework, via a systematic integration of platform, hardware, and algorithm co-design innovations. Despite a growing interest in energy-efficient DNNs, existing techniques lack a systematic optimization across the full stack of design abstraction, from systems through algorithms to hardware implementation. The proposed research advocates an innovative, holistic effort towards energy-efficient and adaptive DNN-powered "IoT cameras" by jointly optimizing the platform-, hardware-, and algorithm-level co-design efforts. On the system level, we will address how to automatically generate and adapt DNN models and implementation, to meet a variety of "IoT devices" application-specific performance needs and device-specific resource constraints. On the hardware level, we will leverage the observed high sparsity in DNN activations for energy-efficient hardware implementations of both DNN training and inference by using low-cost zero predictors and hence bypass unnecessary computations. On the algorithm level, we will develop innovative factorized sparsity regularization in DNN training as well as efficient, controllable adaptive inference mechanisms, fully complementing and closely integrating with our hardware innovations. The proposed research will advance the scientific domain of each level, from system and algorithm, to hardware and a holistic, systematic cross-level methodology for designing energy-efficient intelligent systems. Progress on this project will enable ubiquitous DNN-powered intelligent functions in a significantly increased number of resource-constrained daily-life devices, across numerous camera-based Internet-of-Things (IoT) applications such as traffic monitoring, self-driving and smart cars, personal digital assistants, surveillance and security, and augmented reality. As camera-based IoT devices penetrate all walks of life, by enabling DNN-powered intelligence to be pervasive in these devices, the proposed research can have a tremendous impact on global societies and economies. The research will be integrated with education on energy efficient deep learning. Educational activities include curriculum development, undergraduate research, and outreach to K-12 students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.