CNS Core: Small: Software-Defined Video Analytics Pipeline: Enabling Resilient, High-Accuracy, and Resource-Effective Video Analytics

近年来，随着物联网、边缘计算和5G等高带宽接入网络的增长，机器学习和计算机视觉技术取得了重大进展，导致视频分析系统的广泛采用。这类系统在美国和世界各地的主要城市部署摄像头，以支持监控、交通、公共安全、医疗保健、零售和家庭自动化等领域的各种应用。典型的视频分析系统部署包括视频分析管道(VAP)，视频摄像头部署在机场和医院等不同的感兴趣位置，以持续捕获视频流并通过网络(例如5G)将其传输到执行视频分析处理的云服务器。随着网络状况、计算资源可用性以及重要的是捕获的视频帧的内容随着时间的推移而发生变化，VAP需要不断调整，以支持弹性、高精度和资源高效的视频分析应用。近年来提出的大量VAP适配设计忽略了现代摄像机内置的帧/视频处理可配置性，依赖于昂贵的离线/在线剖析，并且仅限于简单的帧/视频适配，例如帧速率调谐和下采样。该项目旨在开发支持软件定义的视频分析管道架构的关键技术，该架构支持使用目前市场上广泛存在的商用可重新配置网络摄像头进行弹性、高精度、资源高效的视频分析。它将开发(1)第一个软件定义的VAP抽象，它向视频分析管道的第一个阶段--摄像机本身--注入“智能”，(2)第一个通过利用可重新配置的摄像机实现VAP的全自动、实时适应的软件体系结构，这有可能显著提高视频分析系统对摄像机周围环境条件变化的适应能力，以及(3)第一个联合调整复杂的摄像机参数以优化共享VAP的多个分析任务的准确性和资源使用的能力，从而降低VAP部署的成本。拟议的研究将具有直接、对视频分析行业的实际影响和巨大的社会影响。(1)拟议的软件定义的VAP架构将为高精度、资源效率高的VAP提供亟需的参考系统设计和实施，从而最大限度地利用现代网络摄像机的内置帧处理能力，从而有可能促进视频分析系统部署中“智能”摄像机的普及和广泛采用。(2)为实现弹性、高精度、资源效率和成本效益的VAP而开发的技术将促进许多重要的社会VAP应用的广泛采用，如交通、娱乐、医疗保健、零售、汽车、家庭自动化、安全和安保。(3)从技术上讲，这项工作将通过开发通用的软件定义的体系结构来优化其他类别的遥感系统和基于智能传感器(如激光雷达和超宽带传感器)的应用，从而对优化视频分析系统领域产生深远的影响。研究团队将积极向视频分析行业传播和转移开发的技术，并帮助在全球范围内组织一年一度的IEEE自主无人机(UAV)竞赛。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Significant progress in machine learning and computer vision techniques along with growth in Internet of Things, edge computing and high-bandwidth access networks such as 5G in recent years have led to the wide adoption of video analytics systems. Such systems deploy cameras in major cities in the US and around the world to support diverse applications in surveillance, transportation, public safety, health-care, retail, and home automation. A typical video analytics system deployment consists of a video analytics pipeline (VAP), where video cameras are deployed at different locations of interest such as airports and hospitals to continuously capture video streams and transport them over the network (e.g., 5G) to the cloud servers that perform video analytics processing. As the network condition, compute resource availability, and importantly the content of the captured video frames undergo changes over time, the VAP needs to be continuously adapted in order to support resilient, high-accuracy and resource-efficient video analytics applications. The large amount of proposed VAP adaptation design in recent years ignore the built-in frame/video processing configurability of modern cameras, rely on costly offline/online profiling, and are limited to simple frame/video adaptations such as frame rate tuning and down-sampling. This project aims to develop key technologies that enable a software-defined video analytics pipeline architecture that supports resilient, high-accuracy, resource-efficient video analytics using commodity reconfigurable network cameras widely available in the market today. It will develop (1) the first software-defined VAP abstraction that instills “intelligence” into the very first stage of a video analytics pipeline, the camera itself, (2) the first software architecture that enables fully automated, real-time adaptation of VAPs by exploiting reconfigurable cameras, which has the potential to significantly improve the resilience of video analytics systems to environmental condition changes around the camera, and (3) the first capability to jointly adapt complex camera parameters to optimize the accuracy and resource usage of multiple analytics tasks that share a VAP and hence its camera capture, which lowers the cost of VAP deployment.The proposed research will have direct, practical implications to the video analytics industry and large societal impact. (1) The proposed software-defined VAP architecture will provide a much needed reference system design and implementation of high-accuracy, resource-efficient VAPs that maximally exploit the in-built frame processing capabilities of modern network cameras, and thus has the potential to foster the proliferation and wide adoption of “smart” cameras in video analytics system deployment. (2) The technologies developed for enabling resilient, high-accuracy, resource-efficient and cost-efficient VAPs will foster wide adoption of many important societal VAP applications such as transportation, entertainment, health-care, retail, automotive, home automation, safety, and security. (3) Technically, this work will have a far-reaching impact beyond the area of optimizing video analytics systems by developing general software-defined architectures for optimizing other classes of remote sensing systems and applications based on smart sensors such as LiDARs and UWB sensors. The research team will actively disseminate and transfer the technologies developed to the video analytics industry, and help organize the annual IEEE Autonomous Unmanned Aerial Vehicles (UAV) Competition for high school students world-wide.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.