CDSE: Collaborative: Cyber Infrastructure to Enable Computer Vision Applications at the Edge Using Automated Contextual Analysis

数码相机被部署为网络边缘设备，为自动驾驶、交通分析和野生动物观察等任务收集视觉数据。分析大量的视觉数据是一项挑战。现有的计算机视觉方法需要快速的计算机，这超出了许多边缘设备的计算能力。该项目旨在提高计算机视觉方法的效率，使其能够在电池供电的边缘设备上运行。基于视觉数据和补充元数据（例如，地理位置、当地时间），该项目首先提取上下文信息（例如，城市街道在高峰时间预计是忙碌的）。上下文信息可以帮助确定分析结果是否正确。例如，野生动物不应该出现在城市街道上。此外，上下文信息可以提高效率。 仅需要分析某些像素（道路上的像素对于检测汽车有用，而天空中的像素则不有用），这可以显著减少计算量，从而实现对边缘设备的分析。该项目构建了三种服务的网络基础设施：（1）理解上下文信息以减少分析方法的搜索空间，（2）通过只考虑必要的像素来减少计算，（3）基于上下文信息自动评估分析结果，而无需人工。理解上下文信息通过使用背景分割，GPS位置相关逻辑和图像深度图来实现。 背景分析利用随时间的语义分割和分析来识别背景像素，然后通过背景-隐含-前景关系生成推理规则。如果一个像素在很长一段时间内始终被相同的语义标签标记，则该像素被分类为背景像素。背景信息可以推断某些类型的前景对象。例如，如果背景是城市街道，则前景对象可以是车辆或行人;如果检测到野牛，则这可能是错误的。该项目通过向神经网络层添加遮罩来仅处理前景像素。掩蔽卷积可以大大减少计算量，而不会损失精度，并且不需要额外的训练。同时，分层神经网络可以根据上下文跳过模型的部分。例如，天空中的像素只需要由对飞机进行分类的层次节点进行处理。该项目提供了一种在线服务，可以接受输入数据和分析程序，用于自动评估程序，而无需人工创建标签。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估。

Digital cameras are deployed as network edge devices, gathering visual data for such tasks as autonomous driving, traffic analysis, and wildlife observation. Analyzing the vast amount of visual data is a challenge. Existing computer vision methods require fast computers that are beyond the computational capabilities of many edge devices. This project aims to improve the efficiency of computer vision methods so that they can run on battery-powered edge devices. Based on the visual data and complementary metadata (e.g., geographical location, local time), the project first extracts contextual information (such as a city street is expected to be busy at rush hour). The contextual information can help assist determine whether analysis results are correct. For example, a wild animal is not expected on a city street. Moreover, contextual information can improve efficiency. Only certain pixels need to be analyzed (pixels on the road are useful for detecting cars, while pixels in the sky are not) and this can significantly reduce the amount of computation, thus enabling analysis on edge devices. This project constructs a cyberinfrastructure for three services: (1) understand contextual information to reduce the search space of analysis methods, (2) reduce computation by considering only necessary pixels, and (3) automate evaluation of analysis results based on the contextual information without human effort.Understanding contextual information is achieved by using background segmentation, GPS-location-dependent logic, and image depth maps. Background analysis leverages semantic segmentation and analysis over time to identify the background pixels and then generate inference rules via a background-implies-foreground relationship. If a pixel is consistently marked by the same semantic label across a long period of time, this pixel is classified as a background pixel. The background information can infer certain types of foreground objects. For example, if the background is city streets, the foreground objects can be vehicles or pedestrians; if a bison is detected, this is likely a mistake. This project processes only the foreground pixels by adding masks to the neural network layers. Masking convolution can substantially reduce the amount of computation with no loss of accuracy and no additional training is needed. Meanwhile, hierarchical neural networks can skip sections of a model based on context. For example, pixels in the sky only need to be processed by the hierarchy nodes that classify airplanes. The project provides an online service that can accept input data and analysis programs for automatic evaluation of the programs, without human created labels. The evaluation is based on the correlations of background and foreground objects.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.