CNS Core: Small: Software-Defined Video Analytics Pipeline: Enabling Resilient, High-Accuracy, and Resource-Effective Video Analytics

近年来，随着物联网、边缘计算和5G等高带宽接入网络的发展，机器学习和计算机视觉技术取得了重大进展，视频分析系统得到了广泛采用。这些系统在美国和世界各地的主要城市部署了摄像头，以支持监控、交通、公共安全、医疗保健、零售和家庭自动化等领域的各种应用。典型的视频分析系统部署由视频分析管道（VAP）组成，其中视频摄像机部署在机场和医院等不同位置，以连续捕获视频流，并通过网络（例如5G）将其传输到执行视频分析处理的云服务器。随着网络条件、计算资源的可用性，以及重要的是捕获视频帧的内容随着时间的推移而发生变化，VAP需要不断调整，以支持弹性、高精度和资源高效的视频分析应用程序。近年来提出的大量VAP自适应设计忽略了现代摄像机内置的帧/视频处理可配置性，依赖于昂贵的离线/在线分析，并且仅限于简单的帧/视频自适应，如帧速率调整和降采样。该项目旨在开发关键技术，使软件定义的视频分析管道架构能够使用当今市场上广泛使用的商品可重构网络摄像机，支持弹性、高精度、资源高效的视频分析。它将开发(1)第一个软件定义的VAP抽象，将“智能”灌输到视频分析管道的第一阶段，即摄像机本身；(2)第一个软件架构，通过利用可重构摄像机实现全自动、实时适应VAP，这有可能显著提高视频分析系统对摄像机周围环境条件变化的弹性。(3)首次能够共同适应复杂的相机参数，以优化共享VAP的多个分析任务的准确性和资源使用，从而降低VAP部署的成本。所提出的研究将对视频分析行业和巨大的社会影响产生直接的实际影响。(1)提出的软件定义VAP架构将提供一个非常需要的参考系统设计和实现高精度、资源高效的VAP，最大限度地利用现代网络摄像机的内置帧处理能力，因此有可能促进“智能”摄像机在视频分析系统部署中的普及和广泛采用。(2)为实现弹性、高精度、资源高效和成本高效的VAP而开发的技术将促进许多重要的社会VAP应用的广泛采用，如交通、娱乐、医疗保健、零售、汽车、家庭自动化、安全和安保。(3)从技术上讲，这项工作将通过开发通用软件定义架构来优化其他类别的遥感系统和基于智能传感器（如lidar和UWB传感器）的应用，从而对优化视频分析系统领域产生深远的影响。研究团队将积极向视频分析行业传播和转让所开发的技术，并帮助组织面向全球高中生的年度IEEE自主无人驾驶飞行器（UAV）竞赛。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Significant progress in machine learning and computer vision techniques along with growth in Internet of Things, edge computing and high-bandwidth access networks such as 5G in recent years have led to the wide adoption of video analytics systems. Such systems deploy cameras in major cities in the US and around the world to support diverse applications in surveillance, transportation, public safety, health-care, retail, and home automation. A typical video analytics system deployment consists of a video analytics pipeline (VAP), where video cameras are deployed at different locations of interest such as airports and hospitals to continuously capture video streams and transport them over the network (e.g., 5G) to the cloud servers that perform video analytics processing. As the network condition, compute resource availability, and importantly the content of the captured video frames undergo changes over time, the VAP needs to be continuously adapted in order to support resilient, high-accuracy and resource-efficient video analytics applications. The large amount of proposed VAP adaptation design in recent years ignore the built-in frame/video processing configurability of modern cameras, rely on costly offline/online profiling, and are limited to simple frame/video adaptations such as frame rate tuning and down-sampling. This project aims to develop key technologies that enable a software-defined video analytics pipeline architecture that supports resilient, high-accuracy, resource-efficient video analytics using commodity reconfigurable network cameras widely available in the market today. It will develop (1) the first software-defined VAP abstraction that instills “intelligence” into the very first stage of a video analytics pipeline, the camera itself, (2) the first software architecture that enables fully automated, real-time adaptation of VAPs by exploiting reconfigurable cameras, which has the potential to significantly improve the resilience of video analytics systems to environmental condition changes around the camera, and (3) the first capability to jointly adapt complex camera parameters to optimize the accuracy and resource usage of multiple analytics tasks that share a VAP and hence its camera capture, which lowers the cost of VAP deployment.The proposed research will have direct, practical implications to the video analytics industry and large societal impact. (1) The proposed software-defined VAP architecture will provide a much needed reference system design and implementation of high-accuracy, resource-efficient VAPs that maximally exploit the in-built frame processing capabilities of modern network cameras, and thus has the potential to foster the proliferation and wide adoption of “smart” cameras in video analytics system deployment. (2) The technologies developed for enabling resilient, high-accuracy, resource-efficient and cost-efficient VAPs will foster wide adoption of many important societal VAP applications such as transportation, entertainment, health-care, retail, automotive, home automation, safety, and security. (3) Technically, this work will have a far-reaching impact beyond the area of optimizing video analytics systems by developing general software-defined architectures for optimizing other classes of remote sensing systems and applications based on smart sensors such as LiDARs and UWB sensors. The research team will actively disseminate and transfer the technologies developed to the video analytics industry, and help organize the annual IEEE Autonomous Unmanned Aerial Vehicles (UAV) Competition for high school students world-wide.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.