Hierarchical models for the recognition of human activities in video data

随着每天录制和分发的视频数据越来越多，对自动化处理的需求也越来越大。为了解决这些数据的复杂性，基于视频的动作识别需要从只有一个明确定义的活动的预分割片段的简单分类推进到更长的视频序列的分析。已经有了处理这些序列识别的第一个方法，但它们通常考虑严格的时间线，而不考虑人类活动的具体等级性质。提出的项目通过专注于视频序列中人类活动的时间层次分析来填补这一空白。假设人类活动是由基本的构建块组成的，这些构建块可以分为几个阶段，形成更大的、有意义的活动。在此背景下，本研究旨在探索基于视频的动作识别的分层时间结构，目的是分析和识别视频中复杂的人类活动。为了将层次模型转移到真实的动作识别场景中，提出了一个三阶段的方法。首先，构建基于时间小实体的自下而上的人类行为识别系统。实体将在几个时间层上连接和汇集，以构建高级表示。该系统将建立在生成模型上，因为它们已经成功地应用于类似问题的背景下。其次，为了避免标注数据的任务，将实施和评估半监督和无监督的培训程序。因此，现有的未标记训练材料将以半监督或无监督的方式进行分割和聚类，所得单元将构成自动生成语法和语言格式的输入。生成的训练过程应该能够将给定的训练集分割成小部分，通过聚类将它们组合起来，并通过基于定义的实体生成语法来构建活动域的总体表示。第三，随着时间的推移，系统提供了生成的、时间的建模，这些属性将被利用，因为它在整合上下文知识方面的潜力：整体模型的生成性质允许在识别过程的任何阶段以概率分布的形式轻松整合上下文，时间建模不仅提供了上下文的集成，而且还提供了上下文的评估，例如，随着时间的推移，以对象状态的形式。整个最终系统应提供关于环境背景的人类行为的分层识别和分析，以及将该模型应用于各种不同数据集和应用领域所需的训练程序。我们希望该系统将提供新的方法来处理随着时间的推移分析复杂活动的挑战，并允许在这一领域中有新的应用。

With the growing amount of video data recorded and distributed everyday there is also a growing need for automated processing. To address the complexity of these data, video-based action recognition needs to advance from simple classification of pre-segmented clips with only one clearly defined activity towards the analysis of longer video sequences. First approaches to deal with the recognition of those sequences have been made, but they usually consider stringent timelines without regarding the specific hierarchical nature of human activities. The proposed project fills this gap by focusing on the analysis of temporal hierarchies of human activities in video sequences. It is assumed that human activities are made up of basic building blocks that can be subsumed over several stages to form larger, meaningful activities. In this context, this work aims at the exploration of hierarchical temporal structures for video-based action recognition with the goal is to analyze and recognize complex human activities in videos. To transfer hierarchical models to real action recognition scenarios, a three-stage approach is proposed. First, a bottom-up recognition system for human actions based on small temporal entities will be built. The entities will be concatenated and pooled over several temporal layers to build a high-level representation. The system will be built on generative models, as they have been successfully applied in the context of similar problems. Second, to avoid the task of labeling data, semi- and unsupervised training procedures will be implemented and evaluated. Therefore, existing unlabeled training material will be segmented and clustered, either in a semi- or unsupervised way and the resulting units will form the input for an automatically generated grammar and language format. The resulting training procedure should be able to segment a given training set into small parts, to combine them by clustering, and to build an overall representation of the activity domain by the generation of a grammar based on the defined entities. Third, as the system provides for generative, temporal modeling over time, those properties will be exploited with regard to its potential in integrating context knowledge: the generative nature of the overall model allows easy integration of context in the form of probability distributions at any stage of the recognition process and the temporal modeling provides not only an integration of context but also the assessment of context, e.g. in the form of object states, over time. The overall final system should provide both hierarchical recognition and analysis of human actions with regard to environmental context as well as the training routines needed to apply this model to a large variety of different datasets and application domains. We hope that the system will provide new ways to deal with the challenges of analyzing complex activities over time and that it will allow new applications in this field.