Volumetric Features for Large-Scale Video Processing

技术概述。该项目致力于设计从视频流中提取的特征，以实现视觉事件的检测和分类。该项目的重点是从训练数据中无监督地学习特征和分类器，以及有效处理大量数据的能力。仔细设计这些功能至关重要，以确保它们可以高效地处理大量数据，同时准确地提取与应用程序相关的事件。该项目致力于开发能够高效、准确地从视频流中提取的时空特征。这与基于主动存储的分布式搜索体系结构领域的最新发展相结合，该体系结构是专门为处理非常大的图像和视频数据库而设计的，非常适合于有效地利用视频特征检测器。该项目的主要目标包括将特征提取方法与当前的主动存储方法相结合，并在视频检索、监控和法医视频重建等应用的背景下对生成的系统进行评估。该项目解决了与视频流分析相关的基本问题：(1)什么是好的特征表示，是否有单一的表示方式？(2)视频中存在哪些时空基元？(3)如何有效地检测时空基元？在项目过程中，通过(1)开发了一种新的方法来自动分割在外观和运动上都一致的时空区域；以及(2)开发了一种基于直接在时空立方体上操作的新的“体积”算子来提取时空特征并将这些特征用于事件分类的新方法来解决这些问题。分割算法是经典Mean Shift的扩展，体积算子是Box算子的时空扩展，后者已成功地在2D图像中提取和分类特征。这两个发展为未来的视频分析系统提供了基本的模块。这些技术是视频中有效识别对象的重要构建块。除了在算法和功能设计方面的基本贡献外，该项目的范围还包括通过演示这些方法与分布式搜索系统中新兴方法的集成来验证这些方法，这些系统使用主动存储来实现对100TB大小视频集合的高效处理。近年来，数字视频数据量呈指数级增长，原因是数字消费类摄像机的价格越来越低，视频监控系统的大规模部署，数字内容创建的简便性，以及高速网络和大容量存储设备的可用性。手动组织和注释这些内容正变得不可行。不幸的是，搜索、索引和检索视频内容的技术未能跟上步伐。特别是，处理非常大量的视频数据需要高效的方法来对视频进行预处理以提取与感兴趣的时间事件相对应的特征。该项目通过提供能够开发大规模视频分析工具的新技术，直接满足了这一需求。该项目的产品几乎对所有视频分析应用程序都有潜在的影响。该项目侧重于在改进信息获取和安全方面具有重大社会影响的几大类应用程序。特别是，该项目有助于视频检索(例如，用于教育)、视频监控(例如，用于国土安全)、法医视频重建(例如，用于执法)和智能环境(例如，用于企业和家庭)。为了协助将该技术应用于这些领域，该项目包括一项通过外部合作伙伴转让的计划，该外部合作伙伴的作用是提供数据集(例如，IRP发言者视频数据集)、情景、计算资源、评估算法的软件和指导以及使用最先进的主动存储技术。这一合作将使我们的视频处理元素与主动存储技术的新发展相结合，并展示我们的方法对大规模分布式视频分析系统的适用性。网址：http://www.cs.cmu.edu/~hebert/vol3d.html

Technical Overview. This project addresses the design of features extracted from video streams to enable detection and classification of visual events. The emphasis of the project is on unsupervised learning of features and classifiers from training data and on the ability to deal efficiently with large volumes of data. Careful design of the features is crucial to ensure that they can handle a large volume of data efficiently while, at the same time, accurately extracting events that are relevant to the applications. This project focuses on the development of spatio-temporal features that can be efficiently and accurately extracted from video streams. This is combined with recent developments in the area of architectures for distributed search based on active storage, which are specifically designed for processing very large databases of images and videos, are ideally suited for making effective use of the video feature detectors. Key objectives of the project include integrating the feature extraction approach with current active storage approaches and evaluating the resulting systems in the context of applications such as video retrieval, surveillance, and forensic video reconstruction.The project addresses the fundamental questions related to the analysis of video streams: (1) What makes a good feature representation and is there a single choice of representation? (2) What spatio-temporal primitives exist in video? (3) How to efficiently detect spatio-temporal primitives? These questions are addressed in the course of the project by (1) developing a novel approach for the automatic segmentation of spatio-temporal regions that are consistent in both appearance and motion; and (2) developing a novel approach for extracting spatio-temporal features based on new "volumetric" operators that operate directly on the spatio-temporal cube and for using these features for event classification. The segmentation algorithm is an extension of the classical mean shift, and the volumetric operators are spatio-temporal extensions of box operators that have been successful in extracting and classifying features in 2D images. These two developments provide the fundamental modules for future video analysis systems. These techniques are important building blocks for efficient object recognition in video. In addition to the fundamental contributions in algorithms and feature design, the scope of the project includes validating the approaches by demonstrating their integration with emerging approaches in distributed search systems that employ active storage to enable efficient processing of 100 terabyte-sized video collections.Broader Impact. The amount of digital video data has grown exponentially in recent years due to the increasing affordability of digital consumer video cameras, large-scale deployment of video surveillance systems, ease of digital content creation, and availability of high-speed networks and high-capacity storage devices. Manual organization and annotation of this content is becoming infeasible. Unfortunately, the technology for searching, indexing and retrieving video content has failed to keep pace. In particular, the processing of very large volumes of video data requires efficient ways of pre-processing the videos to extract features corresponding to interesting temporal events. The project addresses this need directly by providing new technology that will enable the development of large-scale video analysis tools. The products of project have potential impact on all virtually all applications of video analysis. The project focuses on a few broad classes of applications with substantial societal impact in the areas of improved access to information and security. In particular, the project contributes to video retrieval (e.g., for education), video surveillance (e.g., for homeland security), forensic video reconstruction (e.g., for law enforcement) and smart environments (e.g., for business and homes). In order to assist in the application of the technology to these areas, the project includes a plan for transfer through an external partner whose role is to provide data sets (e.g., the IRP speaker video dataset), scenarios, computing resources, software and guidance for the evaluation of the algorithms, as well as access to state of the art active storage technology. This collaboration will enable the integration our the video processing elements with new developments in active storage technology, and to demonstrate the applicability of our approach to large-scale distributed video analysis systems.URL: http://www.cs.cmu.edu/~hebert/vol3d.html