CAREER: Geometry, Physics and Semantics from Motion: Learning Expressive and Space-Aware Video Representations

该项目为视觉识别、机器人控制和支持场景理解的语言基础开发了视图不变的3D视觉表示。该项目最大限度地减少了有效的3D视觉识别所需的人工注释工作。该项目将在视觉、语言和控制方面注入常识和可负担性推理。它还将介绍视觉运动表征的学习范式，由体现、互动和人类演示和叙述监督，就像人类学习一样。该项目将有助于控制任何具有视觉功能的移动代理，如地面车辆和无人机，使人工智能系统在视觉推理方面更接近人类的表现水平。它将进一步建立人工智能研究与计算神经科学和认知心理学之间的联系，通过提出类似于人类的学习范式，由体现和预测驱动，并通过探索归纳偏差，例如运动/外观解纠缠，需要集成到当前的计算模型中，通过适当的训练，实现人类能够进行的推理类型。该项目的研究将与研究者的教育计划相结合，并将研究结果传播给研究团体。本研究引入了将RGB和RGB- d流分解为摄像机和物体的场景外观和运动的视觉特征表示。外观编码随时间持续存在的属性，例如语义、材料属性、形状等，而运动编码随时间快速变化的属性，例如相机运动、对象位置和姿态以及对象非刚性变形。该项目设想了装备有摄像头的实体代理来观察世界，末端执行器与之互动，学习将他们的视觉运动体验提炼成场景外观的3D特征表示和他们的时间动作条件动态。新的视频表示学习编码对象属性和空间常识，如世界对象大小、3D范围、形状、语义、材料属性、对象持久性，通过优化自监督目标的视图预测、时间框架预测和动作条件预测。这些表示可以根据物体、它们的时间姿态和3D变形轨迹来处理视频流，而不会在遮挡期间产生交叉物体干扰。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project develops view-invariant 3D visual representations for visual recognition, robot control and language grounding that support scene understanding. The project minimizes human annotation efforts required for effective 3D visual recognition. The project will inject common sense and affordability reasoning in vision, language and control. It will also introduce learning paradigms for visuomotor representations supervised by embodiment, interaction and human demonstrations and narrations, just as humans learn. The project will be instrumental in controlling any vision-enabled mobile agents, such as ground vehicles and drones, to bring AI systems closer to the levels of human performance in visual reasoning. It will further establish connections between AI research and computational neuroscience and cognitive psychology by suggesting learning paradigms similar to those of humans, powered by embodiment and prediction, and by exploring inductive biases, such as motion/appearance disentanglement that need to be integrated to current computational models to enable the type of reasoning humans are capable of, with the appropriate amount of training. The research of this project with be integrated with the educational program of the investigator and results of this research will be disseminated to research communities.This research introduces visual feature representations that decompose RGB and RGB-D streams into scene appearance and motion for the camera and the objects. Appearance encodes properties that persist over time, such as semantics, material properties, shape, and so on, and motion encodes properties that vary quickly over time, such as camera motion, object locations and poses, and object non-rigid deformations. The project envisions embodied agents equipped with cameras to observe the world and end-effectors to interact with it, that learn to distill their visuomotor experiences into 3D feature representations of the scene appearance and their temporal action-conditioned dynamics. The new video representations learn to encode object properties and spatial common sense, such as world object size, 3D extent, shape, semantics, material properties, object permanence, by optimizing self-supervised objectives of view prediction, time frame prediction, and action-conditioned prediction. The representations enable processing a video stream in terms of objects, their temporal pose and deformation trajectories in 3D, without cross-object interference during occlusions.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.