CNS Core: Small: Not All Cameras are Created Equal: Systems Support for Highly Adaptive Video Analytics Pipelines

无处不在的摄像机部署，加上计算机视觉算法的稳步改进，已经产生了各种各样的视频分析应用。用例包括监视、交通调度、灾难响应等。然而，尽管他们的承诺，视频分析部署远未普及。一个关键的原因是，视频分析通常是非常昂贵的：视频是数据密集型的，对网络造成压力，分析通常涉及深度神经网络（dnn）来查询视频，需要大量的计算资源。该项目旨在设计和实现实用的视频分析系统，使其能够最有效地利用端到端计算和网络资源，即跨摄像机，服务器和它们之间的网络。所提出的工作的关键见解是通过利用在资源需求和准确性方面不同的帧转换技术（例如，轻量级帧差异与昂贵的对象检测dnn），自适应地放置分析任务。沿着这些思路，该项目涉及三个协同方向。首先，对现有的框架变换技术进行严格分类，研究其计算成本、潜在的数据减少和对响应精度的影响之间的相关性。其次，它开发了端到端系统，该系统可以自动选择适当的帧转换技术在相机上运行，目标是在给定可用资源的情况下优化响应延迟和准确性。第三，它开发了将自适应视频分析扩展到新兴相机设置的技术，例如，多摄像头，可操纵，能量收集；这些系统依赖于提取相机馈送之间的空间和时间关系来指导资源分配决策。拟议的研究目标是很大一部分人口（考虑到视频分析应用的广度），并提高视频分析部署的可访问性和潜力。开发的系统可以为不同规模的组织提供负担得起的（但有效的）视频分析，使他们能够充分利用可用资源。此外，这项工作激发了以前被认为不切实际的新应用，例如，通过能量收集相机对农村地区进行实时监测。该项目还涉及到拓展工作，以吸引目前在计算机科学领域代表性不足的学生。这些努力的关键是扩大跨系统、网络、机器学习和计算机视觉的视频分析管道的跨学科性质。作为该项目一部分的软件和研究工件在一个公共网站上发布：http://web.cs.ucla.edu/~ravi/adaptive_video_analytics/。该网站定期维护，并包括复制说明和包。项目数据在发布后至少在网站上保存5年，根据公众兴趣延长。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The ubiquity of video camera deployments, coupled with steady improvements in computer vision algorithms, has given rise to a diverse range of video analytics applications. Use cases include surveillance, traffic scheduling, disaster response, and more. Yet despite their promise, video analytics deployments are far from widespread. A key reason is that video analysis is often prohibitively expensive: video is data-intensive, stressing the network, and analysis typically involves Deep Neural Networks (DNNs) to query video, requiring substantial compute resources. This project aims to design and implement practical video analytics systems that can adapt their execution to most efficiently utilize end-to-end compute and network resources, i.e., across cameras, servers, and the networks between them.The key insight underlying the proposed work is to adaptively place analytics tasks by leveraging frame-transforming techniques that are diverse in terms of resource requirements and accuracy, e.g., lightweight frame differencing versus expensive object detection DNNs. Along these lines, the project involves three synergistic directions. First, it rigorously classifies existing frame transforming techniques, investigating the correlation between their computation costs, potential data reduction, and impact on response accuracy. Second, it develops end-to-end systems that can automatically select the appropriate frame transforming technique to run on a camera with the goal of optimizing for response latency and accuracy given the available resources. Third, it develops techniques to extend adaptive video analytics to emerging camera settings, e.g., multi-camera, steerable, energy-harvesting; these systems rely on the extraction of spatial and temporal relationships between camera feeds to guide resource allocation decisions.The proposed research targets a large slice of the population (given the breadth of video analytics applications), and improves both the accessibility and potential of video analytics deployments. The developed systems enable affordable (but effective) video analytics for organizations of different scale, allowing them to make the most of their available resources. Furthermore, the work motivates novel applications that were previously deemed impractical, e.g., real-time monitoring of rural areas via energy-harvesting cameras. The project also involves outreach efforts to attract students from populations currently under-represented in computer science. Key to these efforts is magnifying the interdisciplinary nature of video analytics pipelines which span systems, networks, machine learning, and computer vision.The software and research artifacts designed as part of this project are released on a public website: http://web.cs.ucla.edu/~ravi/adaptive_video_analytics/. The site is regularly maintained and includes replication instructions and packages. Project data are kept on the site for at least 5 years after publication, with extensions based on public interest.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.