Volumetric Features for Large-Scale Video Processing

技术概述。该项目解决了从视频流中提取特征的设计，以实现视觉事件的检测和分类。该项目的重点是从训练数据中对特征和分类器进行无监督学习，以及有效处理大量数据的能力。仔细设计这些特性对于确保它们能够有效地处理大量数据，同时准确地提取与应用程序相关的事件至关重要。本项目的重点是开发能够有效、准确地从视频流中提取的时空特征。这与基于主动存储的分布式搜索架构领域的最新发展相结合，这是专门为处理非常大的图像和视频数据库而设计的，非常适合有效地利用视频特征检测器。该项目的主要目标包括将特征提取方法与当前的主动存储方法相结合，并在视频检索、监控和法医视频重建等应用环境中评估结果系统。该项目解决了与视频流分析相关的基本问题：(1)什么是好的特征表示，是否存在单一的表示选择？(2)视频中存在哪些时空原语？(3)如何有效地检测时空原语？这些问题在项目过程中得到了解决：(1)开发了一种新的方法来自动分割在外观和运动上一致的时空区域；(2)开发了一种基于直接在时空立方体上操作的新的“体积”算子提取时空特征的新方法，并将这些特征用于事件分类。分割算法是对经典均值移位算法的扩展，体积算子是对已经成功提取和分类二维图像特征的盒算子的时空扩展。这两方面的发展为未来的视频分析系统提供了基础模块。这些技术是视频中高效目标识别的重要组成部分。除了在算法和功能设计方面的基本贡献之外，该项目的范围还包括通过展示它们与分布式搜索系统中新兴方法的集成来验证这些方法，这些方法采用主动存储来实现100tb大小的视频集合的有效处理。更广泛的影响。近年来，数字视频数据的数量呈指数级增长，这是由于数字消费者摄像机的可负担性越来越高，视频监控系统的大规模部署，数字内容创作的便利性，以及高速网络和大容量存储设备的可用性。手工组织和注释这些内容变得越来越不可行。不幸的是，搜索、索引和检索视频内容的技术未能跟上时代的步伐。特别是，处理非常大量的视频数据需要有效的方法对视频进行预处理，以提取与有趣的时间事件相对应的特征。该项目通过提供能够开发大规模视频分析工具的新技术，直接解决了这一需求。该项目的产品对视频分析的所有应用都有潜在的影响。该项目侧重于在改善信息获取和安全方面具有重大社会影响的几个广泛类别的应用程序。特别是，该项目有助于视频检索（例如，用于教育）、视频监控（例如，用于国土安全）、法医视频重建（例如，用于执法）和智能环境（例如，用于商业和家庭）。为了协助技术在这些领域的应用，该项目包括一项通过外部合作伙伴的转移计划，该合作伙伴的作用是提供数据集（例如，IRP演讲者视频数据集）、场景、计算资源、软件和算法评估指导，以及访问最先进的主动存储技术。这次合作将使我们的视频处理元素与主动存储技术的新发展相结合，并证明我们的方法对大规模分布式视频分析系统的适用性。URL: http://www.cs.cmu.edu/ ~赫伯特/ vol3d.html

Technical Overview. This project addresses the design of features extracted from video streams to enable detection and classification of visual events. The emphasis of the project is on unsupervised learning of features and classifiers from training data and on the ability to deal efficiently with large volumes of data. Careful design of the features is crucial to ensure that they can handle a large volume of data efficiently while, at the same time, accurately extracting events that are relevant to the applications. This project focuses on the development of spatio-temporal features that can be efficiently and accurately extracted from video streams. This is combined with recent developments in the area of architectures for distributed search based on active storage, which are specifically designed for processing very large databases of images and videos, are ideally suited for making effective use of the video feature detectors. Key objectives of the project include integrating the feature extraction approach with current active storage approaches and evaluating the resulting systems in the context of applications such as video retrieval, surveillance, and forensic video reconstruction.The project addresses the fundamental questions related to the analysis of video streams: (1) What makes a good feature representation and is there a single choice of representation? (2) What spatio-temporal primitives exist in video? (3) How to efficiently detect spatio-temporal primitives? These questions are addressed in the course of the project by (1) developing a novel approach for the automatic segmentation of spatio-temporal regions that are consistent in both appearance and motion; and (2) developing a novel approach for extracting spatio-temporal features based on new "volumetric" operators that operate directly on the spatio-temporal cube and for using these features for event classification. The segmentation algorithm is an extension of the classical mean shift, and the volumetric operators are spatio-temporal extensions of box operators that have been successful in extracting and classifying features in 2D images. These two developments provide the fundamental modules for future video analysis systems. These techniques are important building blocks for efficient object recognition in video. In addition to the fundamental contributions in algorithms and feature design, the scope of the project includes validating the approaches by demonstrating their integration with emerging approaches in distributed search systems that employ active storage to enable efficient processing of 100 terabyte-sized video collections.Broader Impact. The amount of digital video data has grown exponentially in recent years due to the increasing affordability of digital consumer video cameras, large-scale deployment of video surveillance systems, ease of digital content creation, and availability of high-speed networks and high-capacity storage devices. Manual organization and annotation of this content is becoming infeasible. Unfortunately, the technology for searching, indexing and retrieving video content has failed to keep pace. In particular, the processing of very large volumes of video data requires efficient ways of pre-processing the videos to extract features corresponding to interesting temporal events. The project addresses this need directly by providing new technology that will enable the development of large-scale video analysis tools. The products of project have potential impact on all virtually all applications of video analysis. The project focuses on a few broad classes of applications with substantial societal impact in the areas of improved access to information and security. In particular, the project contributes to video retrieval (e.g., for education), video surveillance (e.g., for homeland security), forensic video reconstruction (e.g., for law enforcement) and smart environments (e.g., for business and homes). In order to assist in the application of the technology to these areas, the project includes a plan for transfer through an external partner whose role is to provide data sets (e.g., the IRP speaker video dataset), scenarios, computing resources, software and guidance for the evaluation of the algorithms, as well as access to state of the art active storage technology. This collaboration will enable the integration our the video processing elements with new developments in active storage technology, and to demonstrate the applicability of our approach to large-scale distributed video analysis systems.URL: http://www.cs.cmu.edu/~hebert/vol3d.html