III: Small: A Query System for Rapid Audiovisual Analysis of Large-Scale Video Collections

由于计算机视觉和音频处理的进步，现在可以用关于其视听内容(例如，人、地点、物体、音频记录)的基本信息来自动地对大量视频集合进行注释。然而，由于定义更高级别、复杂的感兴趣事件的挑战，在视频上高效地执行更高级别的分析任务仍然很困难。作为回应，该项目寻求通过设计编程系统来实现更复杂和更高生产率的视频分析，以将基本视频注释组合成视频中更高级别的模式和感兴趣的事件。在所提出的系统中创作的查询可以用作感兴趣的视频事件的直接规范，或者用作自动生成数据标签的机制，该数据标签为后续的模型训练提供监督。拟议的系统将适用于许多视频领域；然而，该项目将与记者和新闻媒体工作人员合作，进行视听分析，衡量近十年来美国有线电视新闻广播的多样性和代表性(自2010年以来超过200 000小时)。具体地说，该项目将创建用于回答以下问题的软件工具：哪些人最常出现在新闻中？在什么情况下？(在采访中？在太阳能板上？)某些人报道哪些主题和故事？除了传播这些分析的结果外，该项目还将制作基于网络的互动工具，使学生和公众能够对有线电视新闻的内容进行自己的多样性分析。该项目的主要技术挑战涉及设计一种新的视频分析系统，以确定视频中的时空模式和感兴趣的事件。受早期多媒体数据库查询系统的启发，该系统将通过将所有视频注释(无论是来自像素、音频还是抄本)表示为视频中连续的时空卷来支持多模式视频分析。用户将通过组成(通过时空关系)和操作更简单的时空注释集合的查询来定义复杂的模式。这些查询的组合性质将允许它们在大型视频集合上快速执行，使分析师能够迭代地概念化视频中的新高级模式、原型并指定视频中的新高级模式。为了降低为大型视频集添加注释的成本，该项目还将利用电视和电影视频流的长期运行特性，培训特定于节目或电影视频内容的低成本模型。该项目将研究使用模型蒸馏(并在连续的在线设置中进行)来训练人脸和对象检测模型，这些模型以比现有方法低一个数量级的运行成本在视频流上保持高精度。作为该项目的一部分开发的所有系统将作为开源软件向公众分发，该项目将包括主办黑客松，以教育学生和更广泛的社区了解它们的使用。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Due to advances in computer vision and audio processing, it is now possible to automatically annotate large video collections with basic information about their audiovisual contents (e.g., people, places, objects, audio transcripts). However, it remains difficult to productively carry out higher level analytics tasks on video because of the challenges of defining higher level, complex events of interest. In response, this project seeks to enable more sophisticated and higher productivity video analysis through the design of a programming system for composing basic video annotations into higher level patterns and events of interest in a video. Queries authored in the proposed system can serve as a direct specification of video events of interest, or as a mechanism for automatically generating data labels that provide supervision for subsequent model training. The proposed systems will be applicable to many video domains; however, the project will feature a collaboration with journalists and news media personnel to conduct an audiovisual analysis that measures diversity and representation in nearly a decade of American cable TV news broadcasts (over 200,000 hours since 2010). Specifically, the project will create software tools for answering questions such as: What individuals appear most often on the news? In what contexts? (in interviews? on panels?) What topics and stories do certain individuals cover? In addition to disseminating the results of these analyses, the project will produce interactive web-based tools that will enable students and the public to perform their own diversity analyses of the contents of cable TV news.The primary technical challenge of the project involves the design of a new video analysis system for defining spatio-temporal patterns and events of interest in video. Inspired by early multimedia database query systems, the system will support multi-modal video analyses by representing all video annotations (whether they result from pixels, audio, or transcripts) as continuous space-time volumes in a video. Users will define complex patterns via queries that compose (via spatio-temporal relations) and manipulate collections of simpler space-time annotations. The compositional nature of these queries will allow them to execute rapidly on large video collections, enabling analysts to iteratively conceptualize, prototype, and specify novel high-level patterns in videos. To reduce the cost of annotating large video collections, the project will also exploit the long running nature of TV and film video streams to train low-cost models that are specific to a show or film's video content. The project will investigate the use model distillation (and do so in a continuous, online setting) to train face and object detection models that maintain high accuracy on a video stream at an order of magnitude lower runtime cost than existing methods. All systems developed as part of the project will be distributed to the public as open source software, and the project will involve hosted hackathons to educate students and broader community about their use.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.