CAREER: Geometry, Physics and Semantics from Motion: Learning Expressive and Space-Aware Video Representations

该项目开发视点不变的3D视觉表示，用于视觉识别、机器人控制和支持场景理解的语言基础。该项目最大限度地减少了有效的3D视觉识别所需的人工注释工作。该项目将在视觉、语言和控制方面注入常识和负担能力推理。它还将介绍由体现、交互以及人类演示和叙述监督的视觉运动表征的学习范式，就像人类学习一样。该项目将有助于控制任何具有视觉功能的移动代理，如地面车辆和无人机，使人工智能系统更接近人类在视觉推理方面的表现水平。它将进一步建立人工智能研究与计算神经科学和认知心理学之间的联系，方法是提出由体现和预测驱动的类似于人类的学习范式，并探索归纳偏差，例如需要整合到当前计算模型中的动作/外表解缠，以实现人类能够进行的推理类型，并进行适当的训练。本项目的研究将与研究者的教育计划相结合，并将研究成果传播到研究界。本研究引入视觉特征表征，将RGB和RGB-D流分解为摄像机和对象的场景外观和运动。外观编码随时间变化的属性，如语义、材质属性、形状等，运动编码随时间快速变化的属性，如相机运动、对象位置和姿势以及对象非刚性变形。该项目设想了配备摄像头观察世界和终端效应器与世界互动的具体化代理，这些代理学习将其视觉运动体验提取为场景外观及其时间动作条件动态的3D特征表示。新的视频表示通过优化视图预测、时间帧预测和动作条件预测的自我监督目标，学习编码对象属性和空间常识，例如世界对象大小、3D范围、形状、语义、材料属性、对象持久性。这些表示能够在3D中处理视频流，它们的时间姿势和变形轨迹，而不会在遮挡期间发生跨对象干扰。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project develops view-invariant 3D visual representations for visual recognition, robot control and language grounding that support scene understanding. The project minimizes human annotation efforts required for effective 3D visual recognition. The project will inject common sense and affordability reasoning in vision, language and control. It will also introduce learning paradigms for visuomotor representations supervised by embodiment, interaction and human demonstrations and narrations, just as humans learn. The project will be instrumental in controlling any vision-enabled mobile agents, such as ground vehicles and drones, to bring AI systems closer to the levels of human performance in visual reasoning. It will further establish connections between AI research and computational neuroscience and cognitive psychology by suggesting learning paradigms similar to those of humans, powered by embodiment and prediction, and by exploring inductive biases, such as motion/appearance disentanglement that need to be integrated to current computational models to enable the type of reasoning humans are capable of, with the appropriate amount of training. The research of this project with be integrated with the educational program of the investigator and results of this research will be disseminated to research communities.This research introduces visual feature representations that decompose RGB and RGB-D streams into scene appearance and motion for the camera and the objects. Appearance encodes properties that persist over time, such as semantics, material properties, shape, and so on, and motion encodes properties that vary quickly over time, such as camera motion, object locations and poses, and object non-rigid deformations. The project envisions embodied agents equipped with cameras to observe the world and end-effectors to interact with it, that learn to distill their visuomotor experiences into 3D feature representations of the scene appearance and their temporal action-conditioned dynamics. The new video representations learn to encode object properties and spatial common sense, such as world object size, 3D extent, shape, semantics, material properties, object permanence, by optimizing self-supervised objectives of view prediction, time frame prediction, and action-conditioned prediction. The representations enable processing a video stream in terms of objects, their temporal pose and deformation trajectories in 3D, without cross-object interference during occlusions.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.