RI: Small: Inferring the "Dark Matter" and "Dark Energy" from Image and Video

该项目开发用于提高计算机视觉关键任务性能的核心技术，例如识别物体，理解场景和事件。提高这些任务的性能能够对以下应用产生更广泛的影响：（1）用于安全和及时情报的视频监控;（2）用于灾区救援的智能机器人;以及（3）从无人机拍摄的视频中了解空中场景和活动。在这些应用中，图像中的相当大一部分内容，包括i）实体，例如物体、液体、人类动作和场景;以及ii）关系，例如人类的意图、动作的因果效应、物理场和场景中的吸引力，不能通过当前计算机视觉研究中常用的几何和外观特征来识别。这些实体和关系被称为“暗物质”和“暗能量”，通过类比物理学中的宇宙学模型，并计划开发一个统一的表示，将“可见”和“暗”整合在一个共同的模型中，其中可见光可以用来推断黑暗，而黑暗的姿态约束则用于推断可见光。该研究团队正与工业伙伴合作进行技术转移，具体而言，该项目研究以下主题：i）表示因果知识，超越计算机视觉中的联想知识。偶然模型是人类知识的一大部分，对于回答为什么、为什么不、如果（反事实）会怎样等更深层次的问题至关重要。这项研究是视觉文献中第一次正式研究因果关系（学习，建模和推理）。 ii）推理黑暗实体和关系，以超越当前的几何和基于外观的范式。感知因果关系，人类意图和物理学通常适用于所有类别的对象，场景，动作和事件，即，可跨数据集传输。这些实体和关系比视觉识别中使用的主要特征几何和外观更深刻，更不变。 iii）开发联合表示和联合推理算法。这种联合表示中丰富的上下文和因果关系对于构建强大的视觉系统至关重要，其中每个视觉实体都可以通过多路径推断，但没有系统地研究和集成在现有的范式中。

This project develops core techniques for improving the performance of key tasks in computer vision, such as recognizing objects, understanding scenes and events. Improving the performance of these tasks is able to generate broader impacts to the following applications: (1) video surveillance for security and timely intelligence; (2) intelligent robots for rescue in disaster areas; and (3) aerial scene and activity understanding from videos taken by unmanned aerial vehicles. In these applications, a significant portion of the contents in images, including i) entities such as objects, stuff like liquid, human actions, and scenes; and ii) relations, such as intents of humans, causal effects of actions, physical fields and attractions in a scene, cannot be recognized by the geometry and appearance features that are commonly used in current computer vision research. These entities and relations are referred as the "dark matter" and "dark energy," by analogy to cosmology models in physics, and plans to develop a unified representation that integrate the "visible" and the "dark" in a common model where the visible can be used to infer the dark, and the dark pose constraints for the inference of the visible in return. The research team is collaborating with industrial partner for technology transfer.More specifically, the project studies the following topics: i) Representing causal knowledge to go beyond associational knowledge in computer vision. Casual models are a large part of human knowledge and crucial for answering deeper questions on why, why not, what if (counterfactual). This research is the first formal study of causality (learning, modeling, and reasoning) in the vision literature. ii) Reasoning the dark entities and relations to go beyond the current geometry and appearance-based paradigm. Perceptual causality, human intents and physics are generally applicable to all categories of object, scene, action and events, i.e., transportable across datasets. These entities and relations are deeper, and more invariant, than geometry and appearance - the dominating features used in visual recognition. iii) Developing joint representation and joint inference algorithm. The rich contextual and causal links in this joint representation are essential for building robust vision systems where each visual entity can be inferred through multi-routes, but are not systematically studied and integrated in the existing paradigm.