Hierarchical models for the recognition of human activities in video data

随着每天记录和分发的视频数据越来越多，对自动化处理的需求也越来越大。为了解决这些数据的复杂性，基于视频的动作识别需要从对只有一个明确定义的活动的预分割片段的简单分类发展到对较长视频序列的分析。已经提出了处理这些序列识别的第一批方法，但它们通常考虑严格的时间表，而不考虑人类活动的具体等级性质。提出的项目通过集中分析视频序列中人类活动的时间层次来填补这一空白。假设人类活动是由基本的构件组成的，这些构件可以归结为几个阶段，以形成更大的、有意义的活动。在此背景下，本工作旨在探索基于视频的动作识别的层次化时间结构，目的是分析和识别视频中复杂的人类活动。为了将层次模型转化为真实的动作识别场景，提出了一种三阶段方法。首先，将建立一个基于微小时间实体的自下而上的人类行为识别系统。这些实体将被串联并汇集在几个时间层上，以构建高级表示。该系统将建立在生成性模型的基础上，因为它们已经成功地应用于类似问题。第二，为了避免标记数据的任务，将实施和评估半监督和无监督的训练程序。因此，现有的未标记培训材料将以半监督或无监督的方式进行分段和聚类，所产生的单位将形成自动生成的语法和语言格式的输入。由此产生的训练程序应该能够将给定的训练集分割成小部分，通过聚类将它们组合在一起，并通过基于所定义的实体生成语法来构建活动领域的整体表示。第三，由于该系统随时间提供生成性的时间建模，因此将利用关于其在整合上下文知识方面的潜力的那些属性：整体模型的生成性允许在识别过程的任何阶段以概率分布的形式容易地整合上下文，并且时间建模不仅提供上下文的整合，而且还提供随时间的上下文评估，例如以对象状态的形式。整个最终系统应提供关于环境背景的人类行为的分级识别和分析，以及将该模型应用于各种不同数据集和应用领域所需的培训例程。我们希望，随着时间的推移，该系统将提供新的方法来应对分析复杂活动的挑战，并将允许在这一领域进行新的应用。

With the growing amount of video data recorded and distributed everyday there is also a growing need for automated processing. To address the complexity of these data, video-based action recognition needs to advance from simple classification of pre-segmented clips with only one clearly defined activity towards the analysis of longer video sequences. First approaches to deal with the recognition of those sequences have been made, but they usually consider stringent timelines without regarding the specific hierarchical nature of human activities. The proposed project fills this gap by focusing on the analysis of temporal hierarchies of human activities in video sequences. It is assumed that human activities are made up of basic building blocks that can be subsumed over several stages to form larger, meaningful activities. In this context, this work aims at the exploration of hierarchical temporal structures for video-based action recognition with the goal is to analyze and recognize complex human activities in videos. To transfer hierarchical models to real action recognition scenarios, a three-stage approach is proposed. First, a bottom-up recognition system for human actions based on small temporal entities will be built. The entities will be concatenated and pooled over several temporal layers to build a high-level representation. The system will be built on generative models, as they have been successfully applied in the context of similar problems. Second, to avoid the task of labeling data, semi- and unsupervised training procedures will be implemented and evaluated. Therefore, existing unlabeled training material will be segmented and clustered, either in a semi- or unsupervised way and the resulting units will form the input for an automatically generated grammar and language format. The resulting training procedure should be able to segment a given training set into small parts, to combine them by clustering, and to build an overall representation of the activity domain by the generation of a grammar based on the defined entities. Third, as the system provides for generative, temporal modeling over time, those properties will be exploited with regard to its potential in integrating context knowledge: the generative nature of the overall model allows easy integration of context in the form of probability distributions at any stage of the recognition process and the temporal modeling provides not only an integration of context but also the assessment of context, e.g. in the form of object states, over time. The overall final system should provide both hierarchical recognition and analysis of human actions with regard to environmental context as well as the training routines needed to apply this model to a large variety of different datasets and application domains. We hope that the system will provide new ways to deal with the challenges of analyzing complex activities over time and that it will allow new applications in this field.