CAREER: Learning to Perceive the Interactive 3D World from an Image

该奖项部分由2021年美国救援计划法案（公法117-2）资助。从带门的橱柜到带抽屉的梳妆台，人类建造的世界充满了可交互的物体，这些物体的部件可以被人类操纵。为了让智能机器能够在现实环境中理解和帮助人类，它们必须能够从视觉上理解这些物体，特别是在不受约束的现实环境中。这种理解必须包括理解交互发生时的交互，以及识别交互的机会（即，即使橱柜未触碰也可以与之交互）。这些能力超出了当前AI系统的能力，因为它们主要处理模拟引擎等受限环境中的交互式对象。该项目旨在通过结合大规模第一人称视角的人类互动视频演示以及不包括互动的3D模拟器的知识，构建能够学习这些属性的人工智能系统。由于这些交互对象的普遍性和重要性，该项目有可能加强许多其他学科的努力，例如机器人技术或人类辅助技术。与研究相结合的是一项计划，通过研究机会和增强的课程材料，在多个层面上支持和吸引下一代计算机视觉研究人员。本项目旨在通过四个方向来推进对交互对象的视觉理解来实现这一目标。第一个方向旨在建立无约束第一人称视频中清晰物体的详细3D模型。在这种对发音的物理理解的基础上，第二个方向计划通过关于人类如何实现交互以及它可能完成或揭示的场景的信息来增强这种物理理解。第三项努力旨在通过在视频帧之间建立3D关联，让系统关联并从正在进行的交互示例中学习，从而在发音发生之前理解发音。第四个方向将这种对交互对象的理解与生成完整场景的3D理解的目标联系起来，通过赋予世界的3D重建对可能只是在视野之外或暂时遮挡的对象的信念。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in part under the American Rescue Plan Act of 2021 (Public Law 117-2).The human-built world is filled with interactive objects that have parts that can be manipulated by humans, ranging from cabinets with doors to dressers with drawers. In order for intelligent machines to be able to understand and assist humans in realistic settings, they must be able to understand these objects from vision, and especially in unconstrained realistic settings. This understanding must include understanding the interactions as they occur, as well as recognizing the opportunity for interaction (i.e., that a cabinet could be interacted with even when it is untouched). These abilities are beyond the capabilities of current AI systems since these largely deal with interactive objects in restricted settings such as simulation engines. This project aims to build AI systems that can learn these properties by combining knowledge from large-scale first-person-view video demonstrations of interactions by humans as well as from 3D simulators that do not include interaction. The project has the potential to enhance efforts in many other disciplines, for instance robotics or assistive technology for people, due to the ubiquity and importance of these interactive objects. Integrated with the research is a plan to support and engage the next generation of researchers in computer vision at multiple levels via research opportunities and enhanced course materials.This project aims to achieve this goal via four directions that advance the visual understanding of interactive objects. The first direction aims to build detailed 3D models of articulating objects in unconstrained first person-video. Building on this physical understanding of articulation, the second direction plans to enhance this physical understanding with information about how a human would achieve the interaction and what it might accomplish or reveal about the scene. The third effort aims to enable understanding of articulations before they occur by building associations in 3D across frames of a video, letting a system associate and learn from examples of ongoing interactions. The fourth direction connects this understanding of interactive objects with the goal of producing a 3D understanding of the full scene, by endowing 3D reconstructions of the world with beliefs about objects that may be just out of view or temporarily occluded.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.