CDSE: Collaborative: Cyber Infrastructure to Enable Computer Vision Applications at the Edge Using Automated Contextual Analysis

CDSE：协作：使用自动上下文分析在边缘启用计算机视觉应用的网络基础设施

• 批准号: 2104319
• 负责人: George Thiruvathukal
• 金额: $ 17.47万
• 依托单位: Loyola University of Chicago
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-09-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2104319&HistoricalAwards=false
• 关键词: CDSE; Collaborative; Cyber; Infrastructure; Enable

Digital cameras are deployed as network edge devices, gathering visual data for such tasks as autonomous driving, traffic analysis, and wildlife observation. Analyzing the vast amount of visual data is a challenge. Existing computer vision methods require fast computers that are beyond the computational capabilities of many edge devices. This project aims to improve the efficiency of computer vision methods so that they can run on battery-powered edge devices. Based on the visual data and complementary metadata (e.g., geographical location, local time), the project first extracts contextual information (such as a city street is expected to be busy at rush hour). The contextual information can help assist determine whether analysis results are correct. For example, a wild animal is not expected on a city street. Moreover, contextual information can improve efficiency. Only certain pixels need to be analyzed (pixels on the road are useful for detecting cars, while pixels in the sky are not) and this can significantly reduce the amount of computation, thus enabling analysis on edge devices. This project constructs a cyberinfrastructure for three services: (1) understand contextual information to reduce the search space of analysis methods, (2) reduce computation by considering only necessary pixels, and (3) automate evaluation of analysis results based on the contextual information without human effort.Understanding contextual information is achieved by using background segmentation, GPS-location-dependent logic, and image depth maps. Background analysis leverages semantic segmentation and analysis over time to identify the background pixels and then generate inference rules via a background-implies-foreground relationship. If a pixel is consistently marked by the same semantic label across a long period of time, this pixel is classified as a background pixel. The background information can infer certain types of foreground objects. For example, if the background is city streets, the foreground objects can be vehicles or pedestrians; if a bison is detected, this is likely a mistake. This project processes only the foreground pixels by adding masks to the neural network layers. Masking convolution can substantially reduce the amount of computation with no loss of accuracy and no additional training is needed. Meanwhile, hierarchical neural networks can skip sections of a model based on context. For example, pixels in the sky only need to be processed by the hierarchy nodes that classify airplanes. The project provides an online service that can accept input data and analysis programs for automatic evaluation of the programs, without human created labels. The evaluation is based on the correlations of background and foreground objects.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

数码相机作为网络边缘设备部署，为自动驾驶、交通分析、野生动物观察等任务收集视觉数据。分析大量的视觉数据是一个挑战。现有的计算机视觉方法需要快速的计算机，这超出了许多边缘设备的计算能力。该项目旨在提高计算机视觉方法的效率，使它们能够在电池供电的边缘设备上运行。基于视觉数据和补充元数据（如地理位置、当地时间），该项目首先提取上下文信息（如城市街道在高峰时段预计会很繁忙）。上下文信息可以帮助确定分析结果是否正确。例如，野生动物不应该出现在城市街道上。此外，上下文信息可以提高效率。只需要分析特定的像素（道路上的像素对检测汽车有用，而天空中的像素则没有），这可以显着减少计算量，从而可以在边缘设备上进行分析。本项目构建了三个服务的网络基础设施：(1)理解上下文信息，减少分析方法的搜索空间；(2)只考虑必要的像素，减少计算量；(3)基于上下文信息的分析结果自动评估，无需人工。通过使用背景分割、gps位置相关逻辑和图像深度图来理解上下文信息。背景分析利用随着时间推移的语义分割和分析来识别背景像素，然后通过背景-隐含-前景关系生成推理规则。如果一个像素在很长一段时间内始终被相同的语义标签所标记，则该像素被分类为背景像素。背景信息可以推断出前景物体的某些类型。例如，如果背景是城市街道，前景对象可以是车辆或行人；如果检测到野牛，这很可能是一个错误。这个项目只处理前景像素通过添加蒙版到神经网络层。掩蔽卷积可以在不损失精度的情况下大大减少计算量，并且不需要额外的训练。同时，层次神经网络可以根据上下文跳过模型的部分。例如，天空中的像素只需要由分类飞机的层次节点处理。该项目提供了一个在线服务，可以接受输入的数据和分析程序来自动评估程序，而不需要人为创建标签。评估是基于背景和前景对象的相关性。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Why Accuracy is Not Enough: The Need for Consistency in Object Detection

• DOI: 10.1109/mmul.2022.3175239
• 发表时间: 2022-07
• 期刊: IEEE MultiMedia
• 影响因子: 3.2
• 作者: Caleb Tung;Abhinav Goel;Fischer Bordwell;Nick Eliopoulos;Xiao Hu;Yung-Hsiang Lu;G. Thiruvathukal

Tree-Based Unidirectional Neural Networks for Low-Power Computer Vision

• DOI: 10.1109/mdat.2022.3217016
• 发表时间: 2023-06
• 期刊: IEEE Design & Test
• 影响因子: 2
• 作者: Abhinav Goel;Caleb Tung;Nick Eliopoulos;G. Thiruvathukal;Amy Wang;Yung-Hsiang Lu;James C. Davis

Observing Human Mobility Internationally During COVID-19

• DOI: 10.1109/mc.2022.3175751
• 发表时间: 2023-03
• 期刊: Computer
• 影响因子: 2.2
• 作者: Shane Allcroft;M. Metwaly;Zachery Berg;Isha Ghodgaonkar;Fischer Bordwell;Xinxin Zhao;Xinglei Liu;Jiahao Xu;Subhankar Chakraborty;Vishnu Banna;Akhil Chinnakotla;Abhinav Goel;Caleb Tung;Gore Kao;Wei Zakharov;D. Shoham;G. Thiruvathukal;Yung-Hsiang Lu

Efficient Computer Vision for Embedded Systems

嵌入式系统的高效计算机视觉

• DOI: 10.1109/mc.2022.3145677
• 发表时间: 2022
• 期刊: Computer
• 影响因子: 2.2
• 作者: Thiruvathukal, George K.;Lu, Yung-Hsiang
• 通讯作者: Lu, Yung-Hsiang