CNS Core: Small: Software-Defined Video Analytics Pipeline: Enabling Resilient, High-Accuracy, and Resource-Effective Video Analytics

近年来，机器学习和计算机视觉技术的重大进展沿着物联网、边缘计算和高带宽接入网络（如5G）的增长，导致视频分析系统被广泛采用。这些系统在美国和世界各地的主要城市部署摄像头，以支持监控、交通、公共安全、医疗保健、零售和家庭自动化等各种应用。典型的视频分析系统部署由视频分析管道（VAP）组成，其中视频摄像机被部署在不同的感兴趣的位置，例如机场和医院，以连续捕获视频流并通过网络传输它们（例如，5G）到执行视频分析处理的云服务器。由于网络状况、计算资源可用性以及重要的是所捕获视频帧的内容随时间而发生变化，因此需要不断调整VAP，以支持弹性、高精度和资源高效的视频分析应用。近年来提出的大量VAP适配设计忽略了现代相机的内置帧/视频处理可配置性，依赖于昂贵的离线/在线配置，并且限于简单的帧/视频适配，例如帧速率调谐和下采样。该项目旨在开发关键技术，实现软件定义的视频分析管道架构，支持使用当今市场上广泛使用的商品可重新配置网络摄像机进行弹性，高精度，资源高效的视频分析。它将开发（1）第一个软件定义的VAP抽象，将“智能”灌输到视频分析管道的第一阶段，即摄像机本身，（2）第一个软件架构，通过利用可重新配置的摄像机实现VAP的全自动实时适应，这有可能显着提高视频分析系统对摄像机周围环境条件变化的弹性，以及（3）第一个能够联合调整复杂的摄像机参数，以优化共享VAP的多个分析任务的准确性和资源使用，从而优化其摄像机捕获，这降低了VAP部署的成本。所提出的研究将对视频分析行业和巨大的社会影响具有直接的实际意义。(1)拟议的软件定义VAP架构将提供一个非常需要的参考系统设计和实施的高精度，资源效率的VAP，最大限度地利用内置的帧处理能力的现代网络摄像机，从而有可能促进扩散和广泛采用的“智能”摄像机在视频分析系统部署。(2)为实现弹性、高精度、资源高效和成本高效的VAP而开发的技术将促进许多重要的社会VAP应用的广泛采用，例如交通、娱乐、医疗保健、零售、汽车、家庭自动化、安全和安保。(3)从技术上讲，这项工作将产生深远的影响，超越优化视频分析系统的领域，开发通用的软件定义架构，用于优化其他类别的遥感系统和基于智能传感器（如LiDAR和UWB传感器）的应用。该研究团队将积极向视频分析行业传播和转让所开发的技术，并帮助组织面向全球高中生的IEEE自主无人机（UAV）竞赛。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Significant progress in machine learning and computer vision techniques along with growth in Internet of Things, edge computing and high-bandwidth access networks such as 5G in recent years have led to the wide adoption of video analytics systems. Such systems deploy cameras in major cities in the US and around the world to support diverse applications in surveillance, transportation, public safety, health-care, retail, and home automation. A typical video analytics system deployment consists of a video analytics pipeline (VAP), where video cameras are deployed at different locations of interest such as airports and hospitals to continuously capture video streams and transport them over the network (e.g., 5G) to the cloud servers that perform video analytics processing. As the network condition, compute resource availability, and importantly the content of the captured video frames undergo changes over time, the VAP needs to be continuously adapted in order to support resilient, high-accuracy and resource-efficient video analytics applications. The large amount of proposed VAP adaptation design in recent years ignore the built-in frame/video processing configurability of modern cameras, rely on costly offline/online profiling, and are limited to simple frame/video adaptations such as frame rate tuning and down-sampling. This project aims to develop key technologies that enable a software-defined video analytics pipeline architecture that supports resilient, high-accuracy, resource-efficient video analytics using commodity reconfigurable network cameras widely available in the market today. It will develop (1) the first software-defined VAP abstraction that instills “intelligence” into the very first stage of a video analytics pipeline, the camera itself, (2) the first software architecture that enables fully automated, real-time adaptation of VAPs by exploiting reconfigurable cameras, which has the potential to significantly improve the resilience of video analytics systems to environmental condition changes around the camera, and (3) the first capability to jointly adapt complex camera parameters to optimize the accuracy and resource usage of multiple analytics tasks that share a VAP and hence its camera capture, which lowers the cost of VAP deployment.The proposed research will have direct, practical implications to the video analytics industry and large societal impact. (1) The proposed software-defined VAP architecture will provide a much needed reference system design and implementation of high-accuracy, resource-efficient VAPs that maximally exploit the in-built frame processing capabilities of modern network cameras, and thus has the potential to foster the proliferation and wide adoption of “smart” cameras in video analytics system deployment. (2) The technologies developed for enabling resilient, high-accuracy, resource-efficient and cost-efficient VAPs will foster wide adoption of many important societal VAP applications such as transportation, entertainment, health-care, retail, automotive, home automation, safety, and security. (3) Technically, this work will have a far-reaching impact beyond the area of optimizing video analytics systems by developing general software-defined architectures for optimizing other classes of remote sensing systems and applications based on smart sensors such as LiDARs and UWB sensors. The research team will actively disseminate and transfer the technologies developed to the video analytics industry, and help organize the annual IEEE Autonomous Unmanned Aerial Vehicles (UAV) Competition for high school students world-wide.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.