CRII: RI: Towards Large-Scale Recognition and Fine-Grain Analysis of Human Actions: Pulling Actions Out of Context

本计画主要研究影像中的人类动作辨识问题。人类的行为不会孤立地发生，它也不是视频序列中记录的唯一内容。人类动作的视频剪辑还包含许多其他组件，包括背景场景、交互对象、摄像机运动和其他人的活动。其中一些成分是经常与所考虑的动作类别共同出现的上下文元素。该项目开发了将人类行为与共同发生的因素分离的技术，用于对人类行为进行大规模识别和细粒度视觉解释。所开发的技术可以在广泛的领域中有许多实际应用，从人机交互和机器人技术到安全和医疗保健。 本研究发展了一种从情境中明确分解人类行为的方法来识别人类行为。其关键思想是利用人类行为的共轭样本的信息的好处。共辄样本被定义为在上下文上类似于动作样本但不包含动作的视频剪辑。例如，握手样本的共辄样本可以是示出两个人在握手之前彼此接近的视频序列。握手剪辑和它之前的视频序列具有许多相似甚至相同的上下文元素，包括人、背景场景、相机角度和照明条件。唯一能让这两个视频片段区别开来的是人类的实际行动本身。共轭样本为动作样本提供了补充信息;它可以用来抑制上下文无关性并放大动作信号。该项目的具体研究目标包括：（1）为许多动作类收集人类动作样本;（2）开发挖掘和提取共轭人类动作样本的算法;以及（3）开发一个框架，利用共轭样本将动作从上下文中分离出来的好处来学习分类器，以便对人类动作进行大规模识别和细粒度理解。

This project investigates problems of human action recognition in video. A human action does not occur in isolation, and it is not the only thing recorded in a video sequence. A video clip of a human action also contains many other components, including the background scene, the interacting objects, the camera motion, and the activity of other people. Some of these components are contextual elements that frequently co-occur with the category of action in consideration. The project develops technologies that separated human actions from co-occurring factors for large-scale recognition and fine-grain visual interpretation of human actions. The developed technologies can have many practical applications in a wide range of fields, ranging from human computer interaction and robotics to security and health-care. This research develops an approach to human action recognition by explicitly factorizing human actions from context. The key idea is to exploit the benefits of the information from conjugate samples of human actions. A conjugate sample is defined as a video clip that is contextually similar to an action sample, but does not contain the action. For instance, a conjugate sample of a handshake sample can be the video sequence showing two people approaching each other prior to the handshake. The handshake clip and the video sequence preceding it have many similar or even the same contextual elements, including the people, the background scene, the camera angle, and the lighting condition. The only thing that sets these two video clips apart is the actual human action itself. A conjugate sample provides complementary information to the action sample; it can be used to suppress contextual irrelevance and magnify the action signal. The specific research objectives of this project include: (1) collecting human action samples for many action classes; (2) developing algorithms to mine and extract conjugate human action samples; and (3) developing a framework that utilizes the benefits of conjugate samples for separating actions from context to learn classifiers for large-scale recognition and fine-grain understanding of human actions.