RI: Small: Inferring the "Dark Matter" and "Dark Energy" from Image and Video

该项目开发核心技术，以提高计算机视觉关键任务的性能，如识别物体，理解场景和事件。提高这些任务的性能能够对以下应用产生更广泛的影响：(1)视频监控安全和及时情报；(2)用于灾区救援的智能机器人；(3)通过无人机拍摄的视频了解航拍场景和活动。在这些应用中，图像内容的很大一部分，包括i)物体、液体、人类行为和场景等实体；ii)关系，如人类的意图、行为的因果效应、场景中的物理场和吸引力，不能通过当前计算机视觉研究中常用的几何和外观特征来识别。通过类比物理学中的宇宙学模型，这些实体和关系被称为“暗物质”和“暗能量”，并计划开发一种统一的表示，将“可见”和“黑暗”整合在一个共同的模型中，在这个模型中，可见可以用来推断黑暗，而黑暗反过来对可见的推断构成约束。研究小组正在与工业伙伴合作进行技术转让。更具体地说，该项目研究以下主题：i)表示因果知识，超越计算机视觉中的关联知识。随意模型是人类知识的很大一部分，对于回答诸如“为什么”、“为什么不”、“如果”（反事实）等更深层次的问题至关重要。这项研究是视觉文献中第一个对因果关系（学习、建模和推理）的正式研究。ii)推理黑暗实体和关系，超越当前的几何和基于外观的范式。感知因果关系、人类意图和物理通常适用于所有类别的对象、场景、动作和事件，即可跨数据集传输。这些实体和关系比几何和外观——视觉识别中使用的主要特征——更深刻，更不变。iii)开发联合表示和联合推理算法。这种联合表示中丰富的上下文和因果联系对于构建健壮的视觉系统至关重要，其中每个视觉实体都可以通过多种途径进行推断，但没有系统地研究和整合到现有的范式中。

This project develops core techniques for improving the performance of key tasks in computer vision, such as recognizing objects, understanding scenes and events. Improving the performance of these tasks is able to generate broader impacts to the following applications: (1) video surveillance for security and timely intelligence; (2) intelligent robots for rescue in disaster areas; and (3) aerial scene and activity understanding from videos taken by unmanned aerial vehicles. In these applications, a significant portion of the contents in images, including i) entities such as objects, stuff like liquid, human actions, and scenes; and ii) relations, such as intents of humans, causal effects of actions, physical fields and attractions in a scene, cannot be recognized by the geometry and appearance features that are commonly used in current computer vision research. These entities and relations are referred as the "dark matter" and "dark energy," by analogy to cosmology models in physics, and plans to develop a unified representation that integrate the "visible" and the "dark" in a common model where the visible can be used to infer the dark, and the dark pose constraints for the inference of the visible in return. The research team is collaborating with industrial partner for technology transfer.More specifically, the project studies the following topics: i) Representing causal knowledge to go beyond associational knowledge in computer vision. Casual models are a large part of human knowledge and crucial for answering deeper questions on why, why not, what if (counterfactual). This research is the first formal study of causality (learning, modeling, and reasoning) in the vision literature. ii) Reasoning the dark entities and relations to go beyond the current geometry and appearance-based paradigm. Perceptual causality, human intents and physics are generally applicable to all categories of object, scene, action and events, i.e., transportable across datasets. These entities and relations are deeper, and more invariant, than geometry and appearance - the dominating features used in visual recognition. iii) Developing joint representation and joint inference algorithm. The rich contextual and causal links in this joint representation are essential for building robust vision systems where each visual entity can be inferred through multi-routes, but are not systematically studied and integrated in the existing paradigm.