RI: Small: Inferring the "Dark Matter" and "Dark Energy" from Image and Video

该项目开发了提高计算机视觉关键任务性能的核心技术，如识别对象、理解场景和事件。提高这些任务的性能能够对以下应用产生更广泛的影响：(1)用于安全和及时情报的视频监控；(2)用于灾区救援的智能机器人；以及(3)从无人机拍摄的视频中了解空中场景和活动。在这些应用中，图像中的很大一部分内容，包括i)物体、液体、人体动作和场景等实体；以及ii)关系，如人的意图、动作的因果效果、物理场和场景中的吸引力，无法通过当前计算机视觉研究中常用的几何和外观特征来识别。通过类比物理学中的宇宙学模型，这些实体和关系被称为“暗物质”和“暗能量”，并计划开发一种统一的表示法，将“可见”和“暗”整合到一个共同的模型中，其中可见可以用来推断黑暗，而暗则对可见的推断构成限制。该研究团队正在与工业伙伴合作进行技术转移。更具体地说，该项目研究了以下主题：i)在计算机视觉中表示因果知识，超越联想知识。休闲模特是人类知识的一大部分，对于回答为什么、为什么不、如果(反事实)等更深层次的问题至关重要。这项研究是视觉文献中第一个关于因果关系(学习、建模和推理)的正式研究。Ii)对黑暗实体和关系进行推理，以超越当前基于几何和外观的范例。知觉因果、人类意图和物理通常适用于所有类别的对象、场景、动作和事件，即可跨数据集传输。这些实体和关系比几何和外观更深，更不变，而几何和外观是视觉识别中使用的主要特征。3)开发了联合表示和联合推理算法。这种联合表征中丰富的上下文和因果联系对于建立健壮的视觉系统至关重要，在这种系统中，每个视觉实体都可以通过多条路线推断出来，但没有系统地研究和整合到现有的范例中。

This project develops core techniques for improving the performance of key tasks in computer vision, such as recognizing objects, understanding scenes and events. Improving the performance of these tasks is able to generate broader impacts to the following applications: (1) video surveillance for security and timely intelligence; (2) intelligent robots for rescue in disaster areas; and (3) aerial scene and activity understanding from videos taken by unmanned aerial vehicles. In these applications, a significant portion of the contents in images, including i) entities such as objects, stuff like liquid, human actions, and scenes; and ii) relations, such as intents of humans, causal effects of actions, physical fields and attractions in a scene, cannot be recognized by the geometry and appearance features that are commonly used in current computer vision research. These entities and relations are referred as the "dark matter" and "dark energy," by analogy to cosmology models in physics, and plans to develop a unified representation that integrate the "visible" and the "dark" in a common model where the visible can be used to infer the dark, and the dark pose constraints for the inference of the visible in return. The research team is collaborating with industrial partner for technology transfer.More specifically, the project studies the following topics: i) Representing causal knowledge to go beyond associational knowledge in computer vision. Casual models are a large part of human knowledge and crucial for answering deeper questions on why, why not, what if (counterfactual). This research is the first formal study of causality (learning, modeling, and reasoning) in the vision literature. ii) Reasoning the dark entities and relations to go beyond the current geometry and appearance-based paradigm. Perceptual causality, human intents and physics are generally applicable to all categories of object, scene, action and events, i.e., transportable across datasets. These entities and relations are deeper, and more invariant, than geometry and appearance - the dominating features used in visual recognition. iii) Developing joint representation and joint inference algorithm. The rich contextual and causal links in this joint representation are essential for building robust vision systems where each visual entity can be inferred through multi-routes, but are not systematically studied and integrated in the existing paradigm.