CRII: RI: Towards Large-Scale Recognition and Fine-Grain Analysis of Human Actions: Pulling Actions Out of Context

本课题研究视频中的人体动作识别问题。人类行为并不是孤立地发生的，也不是视频序列中唯一记录的事情。人类动作的视频剪辑还包含许多其他组件，包括背景场景、交互对象、摄像机运动和其他人的活动。其中一些构成部分是经常与所考虑的行动类别同时出现的背景要素。该项目开发了将人类行为与共生因素分离的技术，以实现对人类行为的大规模识别和细粒度视觉解释。开发的技术可以在广泛的领域有许多实际应用，从人机交互和机器人到安全和医疗保健。这项研究开发了一种通过明确地将人类行为从背景中分解出来的方法来识别人类行为。关键的想法是利用人类行为的共轭样本中的信息带来的好处。共轭样本被定义为在上下文上类似于动作样本的视频剪辑，但不包含动作。例如，握手样本的共轭样本可以是在握手之前显示两个人彼此靠近的视频序列。握手片段和之前的视频序列具有许多相似甚至相同的上下文元素，包括人物、背景场景、摄像机角度和照明条件。唯一区别这两个视频片段的是实际的人类行为本身。共轭样本为动作样本提供补充信息；它可用于抑制上下文无关并放大动作信号。该项目的具体研究目标包括：(1)收集多个动作类的人类动作样本；(2)开发挖掘和提取共轭人类动作样本的算法；(3)开发一个框架，利用共轭样本将动作与上下文分离的优点来学习分类器，以实现对人类行为的大规模识别和细粒度理解。

This project investigates problems of human action recognition in video. A human action does not occur in isolation, and it is not the only thing recorded in a video sequence. A video clip of a human action also contains many other components, including the background scene, the interacting objects, the camera motion, and the activity of other people. Some of these components are contextual elements that frequently co-occur with the category of action in consideration. The project develops technologies that separated human actions from co-occurring factors for large-scale recognition and fine-grain visual interpretation of human actions. The developed technologies can have many practical applications in a wide range of fields, ranging from human computer interaction and robotics to security and health-care. This research develops an approach to human action recognition by explicitly factorizing human actions from context. The key idea is to exploit the benefits of the information from conjugate samples of human actions. A conjugate sample is defined as a video clip that is contextually similar to an action sample, but does not contain the action. For instance, a conjugate sample of a handshake sample can be the video sequence showing two people approaching each other prior to the handshake. The handshake clip and the video sequence preceding it have many similar or even the same contextual elements, including the people, the background scene, the camera angle, and the lighting condition. The only thing that sets these two video clips apart is the actual human action itself. A conjugate sample provides complementary information to the action sample; it can be used to suppress contextual irrelevance and magnify the action signal. The specific research objectives of this project include: (1) collecting human action samples for many action classes; (2) developing algorithms to mine and extract conjugate human action samples; and (3) developing a framework that utilizes the benefits of conjugate samples for separating actions from context to learn classifiers for large-scale recognition and fine-grain understanding of human actions.