Collaborative Research: OAC Core: Advancing Low-Power Computer Vision at the Edge

合作研究：OAC Core：推进边缘低功耗计算机视觉

• 批准号: 2107230
• 负责人: Yung-Hsiang Lu
• 金额: $ 25万
• 依托单位: Purdue University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-07-01 至 2024-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2107230&HistoricalAwards=false
• 关键词: Collaborative; Research; OAC; Core; Advancing

This proposal enables low-power edge computers, such as mobile phones, drones, and Internet-of-Things devices, to benefit society. Computer vision is the technology to automatically analyze images and videos. Computer vision on these devices can keep humans safe, for example by spotting dangers in a factory or at a construction site. This project addresses two challenges that hamper practical adoption of computer vision on edge devices. The first challenge is that current computer vision approaches require powerful computers, but these computers are too far away and have long response time. This project brings the computers to the places where data is acquired. The project makes computer vision more efficient, so that visual data can be analyzed by small edge devices like phones and drones. The second challenge is that building complex software for computer vision is difficult. This project provides software engineering support for emerging computer vision technologies. As a result of addressing these two challenges, computer vision on the edge can become feasible.Bringing computer vision (CV) to devices on the network edge is an essential component of realizing NSF's goal of distributed cyberinfrastructure. This project makes CV on the edge feasible and enables scientific and engineering innovation through improved response time, reduced need for network coverage, and decreased storage costs. This project solves two critical challenges that hinder the transition of edge-based CV into practice. (1) This project makes CV more efficient and edge-friendly. Current CV techniques (e.g., deep neural networks) assume server-class resources (such as graphics processing units, gigabytes of memory); these resources are not available at the edge. This project reduces the resource requirements needed for CV. The methods consider alternative neural network architectures and eliminate redundancies while processing visual data. This project also develops CV-specific distribution techniques to enable edge devices to collaborate on large vision tasks. (2) This project provides software engineering support for CV technologies. Solving real-world CV problems requires engineering new CV applications, often by re-implementing research model architectures as components in new designs. This project develops a library of exemplary CV model implementations for low-power platforms. These exemplars can be used as high-quality components in new CV applications. The project identifies factors that promote and inhibit the reproducibility of CV models. This project also identifies engineering best practices by surveying and interviewing experts in low-power CV and by studying their errors.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该提案使低功耗边缘计算机（如移动的手机、无人机和物联网设备）能够造福社会。计算机视觉是自动分析图像和视频的技术。这些设备上的计算机视觉可以保护人类的安全，例如通过发现工厂或建筑工地的危险。该项目解决了阻碍边缘设备上实际采用计算机视觉的两个挑战。第一个挑战是，目前的计算机视觉方法需要强大的计算机，但这些计算机距离太远，响应时间长。该项目将计算机带到获取数据的地方。该项目使计算机视觉更有效，因此视觉数据可以通过手机和无人机等小型边缘设备进行分析。第二个挑战是构建复杂的计算机视觉软件很困难。该项目为新兴的计算机视觉技术提供软件工程支持。解决这两个挑战后，边缘计算机视觉将成为可能。将计算机视觉（CV）引入网络边缘设备是实现NSF分布式网络基础设施目标的重要组成部分。该项目使边缘CV变得可行，并通过改善响应时间，减少对网络覆盖的需求和降低存储成本来实现科学和工程创新。该项目解决了阻碍基于边缘的CV向实践过渡的两个关键挑战。(1)该项目使CV更高效和边缘友好。当前的CV技术（例如，深度神经网络）采用服务器级资源（例如图形处理单元、千兆字节的存储器）;这些资源在边缘不可用。该项目减少了CV所需的资源要求。该方法考虑了替代神经网络架构，并在处理视觉数据时消除冗余。该项目还开发了CV特定的分发技术，使边缘设备能够在大型视觉任务上进行协作。(2)该项目为CV技术提供软件工程支持。解决现实世界的CV问题需要设计新的CV应用程序，通常是通过重新实现研究模型架构作为新设计中的组件。该项目开发了一个低功耗平台的示例性CV模型实现库。这些样本可用作新CV应用程序中的高质量组件。该项目确定了促进和抑制CV模型再现性的因素。该项目还通过调查和采访低功耗CV方面的专家以及研究他们的错误来确定工程最佳实践。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Snapshot Metrics Are Not Enough: Analyzing Software Repositories with Longitudinal Metrics

快照指标还不够：使用纵向指标分析软件存储库

• DOI: 10.1145/3551349.3559517
• 发表时间: 2022
• 期刊: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering
• 作者: Synovic, Nicholas M.;Hyatt, Matt;Sethi, Rohan;Thota, Sohini;Shilpika;Miller, Allan J.;Jiang, Wenxin;Amobi, Emmanuel S.;Pinderski, Austin;Läufer, Konstantin
• 通讯作者: Läufer, Konstantin

An Empirical Study of Artifacts and Security Risks in the Pre-trained Model Supply Chain

• DOI: 10.1145/3560835.3564547
• 发表时间: 2022-11
• 期刊: Proceedings of the 2022 ACM Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses
• 作者: Wenxin Jiang;Nicholas Synovic;R. Sethi;Aryan Indarapu;Matt Hyatt;Taylor R. Schorlemmer;G. Thiruvathukal

An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry

• DOI: 10.1109/icse48619.2023.00206
• 发表时间: 2023-03
• 期刊: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)
• 作者: Wenxin Jiang;Nicholas Synovic;Matt Hyatt;Taylor R. Schorlemmer;R. Sethi;Yung-Hsiang Lu;G. Thiruvathukal;James C. Davis

Evolution of Winning Solutions in the 2021 Low-Power Computer Vision Challenge

2021 年低功耗计算机视觉挑战赛获胜解决方案的演变

• 发表时间: 2023
• 期刊: IEEE intelligent systems
• 影响因子: 6.4
• 作者: Hu, X.;Jiao, Z.;Kocher, A.;Wu, Z.;Liu, J.;Davis, J. C.;Thiruvathukal, G. K.;Lu, Y.-H.
• 通讯作者: Lu, Y.-H.

Reusing Deep Learning Models: Challenges and Directions in Software Engineering

• DOI: 10.1109/jva60410.2023.00015
• 发表时间: 2023-07
• 期刊: 2023 IEEE John Vincent Atanasoff International Symposium on Modern Computing (JVA)
• 作者: James C. Davis;Purvish Jajal;Wenxin Jiang;Taylor R. Schorlemmer;Nicholas Synovic;G. Thiruvathukal