CRII: RI: Towards Large-Scale Recognition and Fine-Grain Analysis of Human Actions: Pulling Actions Out of Context

本项目研究视频中人类动作识别的问题。一个人的行为不是孤立发生的，它也不是视频序列中唯一记录的东西。人类动作的视频片段还包含许多其他组件，包括背景场景、交互对象、摄像机运动和其他人的活动。其中一些组件是上下文元素，经常与所考虑的动作类别同时出现。该项目开发了将人类行为与共同发生的因素分离开来的技术，用于大规模识别和对人类行为的细粒度视觉解释。开发的技术可以在广泛的领域有许多实际应用，从人机交互和机器人到安全和保健。本研究开发了一种基于情境的人类行为识别方法。关键思想是利用从人类行为的共轭样本的信息的好处。共轭样本被定义为一个视频剪辑，它在上下文中与动作样本相似，但不包含动作。例如，握手样本的共轭样本可以是显示两个人在握手之前相互接近的视频序列。握手片段和之前的视频序列有许多相似甚至相同的上下文元素，包括人物、背景场景、摄像机角度和照明条件。这两个视频片段的唯一区别是实际的人类行为本身。共轭样例向作用样例提供补充信息；它可以用来抑制上下文无关性和放大动作信号。本项目的具体研究目标包括：(1)收集多个动作类别的人体动作样本；(2)开发挖掘和提取共轭人体动作样本的算法；(3)开发一个框架，利用共轭样本从上下文中分离动作的好处来学习分类器，用于大规模识别和对人类行为的细粒度理解。

This project investigates problems of human action recognition in video. A human action does not occur in isolation, and it is not the only thing recorded in a video sequence. A video clip of a human action also contains many other components, including the background scene, the interacting objects, the camera motion, and the activity of other people. Some of these components are contextual elements that frequently co-occur with the category of action in consideration. The project develops technologies that separated human actions from co-occurring factors for large-scale recognition and fine-grain visual interpretation of human actions. The developed technologies can have many practical applications in a wide range of fields, ranging from human computer interaction and robotics to security and health-care. This research develops an approach to human action recognition by explicitly factorizing human actions from context. The key idea is to exploit the benefits of the information from conjugate samples of human actions. A conjugate sample is defined as a video clip that is contextually similar to an action sample, but does not contain the action. For instance, a conjugate sample of a handshake sample can be the video sequence showing two people approaching each other prior to the handshake. The handshake clip and the video sequence preceding it have many similar or even the same contextual elements, including the people, the background scene, the camera angle, and the lighting condition. The only thing that sets these two video clips apart is the actual human action itself. A conjugate sample provides complementary information to the action sample; it can be used to suppress contextual irrelevance and magnify the action signal. The specific research objectives of this project include: (1) collecting human action samples for many action classes; (2) developing algorithms to mine and extract conjugate human action samples; and (3) developing a framework that utilizes the benefits of conjugate samples for separating actions from context to learn classifiers for large-scale recognition and fine-grain understanding of human actions.