III: Medium: VOCAL: Video Organization and Interactive Compositional AnaLytics

相机部署通常用于许多应用，如交通监控、动物行为跟踪、自动驾驶、土木工程等。从这些视频流中提取价值是一个关键的研究和商业挑战;一个可以组织和提供界面的系统，使用户能够轻松地与大规模视频进行交互和查询，这将在许多商业和学术领域产生变革。然而，开发现代视频应用所需的视频数据管理系统仍处于起步阶段。现有的系统具有限制其实际用途：它们不容易适应新的领域;它们限于不支持询问复杂的查询;并且大多数系统彼此独立地处理来自多个摄像机的视频流，即使摄像机是协调部署的一部分。这个项目通过开发VOCAL来解决这些限制：一个用于视频组织和交互式构图分析的开源系统。VOCAL由一套用于端到端视频分析的领域无关工具组成。它支持用户（1）交互式组织视频数据，（2）表达和执行复杂查询，以及（3）查询多视图摄像机部署。该项目还为本科生和研究生提供研究经验，并为K-12学生提供有关视频管理和分析的教学材料。为了实现上述目标，该项目为数据库，计算机视觉和AI提供了新的方法。它还汇集了跨这些学科的一些独立的努力。特别是，VOCAL强调了使用最近的自监督计算机视觉方法来构建算法的可能性，这些算法可以使数据探索适用于大型视频数据集，从而允许快速开发特定领域的视频事件识别模型。VOCAL还利用场景图表示，允许用户将复杂的查询表示为简单查询的组合。然后，它开发了新的方法，这种查询的交互式规范和高效执行。 最后，VOCAL为无缝查询多视角摄像机部署提供了新的方法。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Camera deployments are commonly used in many applications such as traffic monitoring, animal behavior tracking, autonomous driving, civil engineering, and more. Extracting value from these video streams is a key research and commercial challenge; a system that can organize and provide an interface for users to easily interact with and query large-scale video is poised to be transformative in many commercial and academic domains. Yet, the video data management systems required to develop modern video applications are still in their infancy. Existing systems have important limitations that restrict their practical use: they do not adapt easily to new domains; they have limited to no support for asking complex queries; and most systems process video streams from multiple cameras independently of one another, even if the cameras are part of a coordinated deployment. This project addresses these limitations by developing VOCAL: an open-source system for Video Organization and Interactive Compositional AnaLytics. VOCAL consists of a suite of domain-agnostic tools for end-to-end video analytics. It supports users with (1) interactively organizing video data, (2) expressing and executing complex queries, and (3) querying multi-view camera deployments. This project also provides research experiences for undergraduate and graduate students and produces materials to teach K-12 students about video management and analytics.To meet the above goals, this project contributes new approaches in databases, computer vision, and AI. It also brings together some of the independent efforts across these disciplines. In particular, VOCAL highlights the possibilities of using recent self-supervised computer vision methods to build algorithms that can make data exploration feasible for large video datasets, and thereby, allowing the rapid development of domain-specific video event recognition models. VOCAL also utilizes scene graph representations to allow users to express complex queries as compositions of simpler ones. It then develops new approaches for the interactive specification and efficient execution of such queries. Finally, VOCAL contributes new approaches to seamlessly querying multi-view camera deployments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.