Collaborative Research: Enabling Intelligent Cameras in Internet-of-Things via a Holistic Platform, Algorithm, and Hardware Co-design

协作研究：通过整体平台、算法和硬件协同设计实现物联网中的智能相机

• 批准号: 1934767
• 负责人: Yingyan Lin
• 金额: $ 27.23万
• 依托单位: William Marsh Rice University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-09-15 至 2023-11-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1934767&HistoricalAwards=false
• 关键词: Collaborative; Research; Enabling; Intelligent; Cameras

There has been a tremendous demand for bringing Deep Neural Network (DNN) powered functionality into Internet of Thing (IoT) devices to enable ubiquitous intelligent "IoT cameras". However, state-of-the-art DNNs have a prohibitive energy cost, making them impractical to be deployed in resource-constrained IoT platforms. This project will develop a novel energy-efficient DNN framework, via a systematic integration of platform, hardware, and algorithm co-design innovations. Despite a growing interest in energy-efficient DNNs, existing techniques lack a systematic optimization across the full stack of design abstraction, from systems through algorithms to hardware implementation. The proposed research advocates an innovative, holistic effort towards energy-efficient and adaptive DNN-powered "IoT cameras" by jointly optimizing the platform-, hardware-, and algorithm-level co-design efforts. On the system level, we will address how to automatically generate and adapt DNN models and implementation, to meet a variety of "IoT devices" application-specific performance needs and device-specific resource constraints. On the hardware level, we will leverage the observed high sparsity in DNN activations for energy-efficient hardware implementations of both DNN training and inference by using low-cost zero predictors and hence bypass unnecessary computations. On the algorithm level, we will develop innovative factorized sparsity regularization in DNN training as well as efficient, controllable adaptive inference mechanisms, fully complementing and closely integrating with our hardware innovations. The proposed research will advance the scientific domain of each level, from system and algorithm, to hardware and a holistic, systematic cross-level methodology for designing energy-efficient intelligent systems. Progress on this project will enable ubiquitous DNN-powered intelligent functions in a significantly increased number of resource-constrained daily-life devices, across numerous camera-based Internet-of-Things (IoT) applications such as traffic monitoring, self-driving and smart cars, personal digital assistants, surveillance and security, and augmented reality. As camera-based IoT devices penetrate all walks of life, by enabling DNN-powered intelligence to be pervasive in these devices, the proposed research can have a tremendous impact on global societies and economies. The research will be integrated with education on energy efficient deep learning. Educational activities include curriculum development, undergraduate research, and outreach to K-12 students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

对于将深度神经网络（DNN）驱动的功能引入物联网（IoT）设备以实现无处不在的智能“IoT相机”存在巨大的需求。然而，最先进的DNN具有令人望而却步的能源成本，使得它们在资源受限的物联网平台中部署不切实际。该项目将通过平台、硬件和算法协同设计创新的系统集成，开发一种新型的节能DNN框架。尽管对节能DNN的兴趣越来越大，但现有技术缺乏对整个设计抽象堆栈的系统优化，从系统到算法再到硬件实现。拟议的研究倡导通过联合优化平台、硬件和算法级的协同设计工作，实现节能和自适应DNN驱动的“物联网相机”的创新、整体努力。在系统层面，我们将解决如何自动生成和调整DNN模型和实现，以满足各种“物联网设备”特定于应用的性能需求和特定于设备的资源约束。在硬件层面上，我们将利用DNN激活中观察到的高稀疏性，通过使用低成本零预测器来实现DNN训练和推理的节能硬件实现，从而绕过不必要的计算。在算法层面，我们将在DNN训练中开发创新的因子化稀疏正则化，以及高效、可控的自适应推理机制，与我们的硬件创新充分互补并紧密结合。 拟议的研究将推进每个层次的科学领域，从系统和算法，到硬件和一个整体的，系统的跨层次的方法来设计节能的智能系统。该项目的进展将在大量资源受限的日常生活设备中实现无处不在的DNN智能功能，包括众多基于摄像头的物联网（IoT）应用，如交通监控，自动驾驶和智能汽车，个人数字助理，监控和安全以及增强现实。随着基于摄像头的物联网设备渗透到各行各业，通过使DNN驱动的智能在这些设备中无处不在，拟议的研究可以对全球社会和经济产生巨大影响。 该研究将与节能深度学习教育相结合。 教育活动包括课程开发，本科生研究和K-12学生的推广。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

项目成果

Contrastive quant: quantization makes stronger contrastive learning

• DOI: 10.1145/3489517.3530419
• 发表时间: 2022-07
• 期刊: Proceedings of the 59th ACM/IEEE Design Automation Conference
• 作者: Y. Fu;Qixuan Yu;Meng Li;Xuefeng Ouyang;Vikas Chandra;Yingyan Lin

InstantNet: Automated Generation and Deployment of Instantaneously Switchable-Precision Networks

InstantNet：自动生成和部署即时可切换精度网络

• 发表时间: 2021
• 期刊: The Design Automation Conference
• 作者: Fu, Yonggan;Yu, Zhongzhi;Zhang, Yongan;Jiang, Yifan;Li, Chaojian;Liang, Yongyuan;Jiang, Mingchao;Wang, Zhangyang;Lin, Yingyan
• 通讯作者: Lin, Yingyan

Early-Bird GCNs: Graph-Network Co-optimization towards More Efficient GCN Training and Inference via Drawing Early-Bird Lottery Tickets

• DOI: 10.1609/aaai.v36i8.20873
• 发表时间: 2021-03
• 作者: Haoran You;Zhihan Lu;Zijian Zhou;Y. Fu;Yingyan Lin

2-in-1 Accelerator: Enabling Random Precision Switch for Winning Both Adversarial Robustness and Efficiency

• DOI: 10.1145/3466752.3480082
• 发表时间: 2021-09
• 期刊: MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture
• 作者: Yonggan Fu;Yang Zhao;Qixuan Yu;Chaojian Li;Yingyan Lin

SACoD: Sensor Algorithm Co-Design Towards Efficient CNN-powered Intelligent PhlatCam

• DOI: 10.1109/iccv48922.2021.00512
• 发表时间: 2021-10
• 期刊: 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
• 作者: Yonggan Fu;Yang Zhang;Yue Wang;Zhihan Lv;Vivek Boominathan;A. Veeraraghavan;Yingyan Lin