CAREER: Learning to Perceive the Interactive 3D World from an Image

该奖项的部分资金来自《2021年美国救援计划法案》(公法117-2)。人类建造的世界充满了互动物品，这些物品的部件可以由人类操纵，从带门的橱柜到带抽屉的梳妆台。为了让智能机器能够在现实环境中理解和帮助人类，它们必须能够从视觉上理解这些对象，特别是在不受限制的现实环境中。这种理解必须包括了解交互发生时的情况，以及认识到交互的机会(即，即使柜子未被触碰，也可以与其交互)。这些能力超出了当前人工智能系统的能力，因为它们主要处理受限环境中的交互对象，如模拟引擎。该项目旨在建立人工智能系统，通过结合来自大规模第一人称观看的人类互动视频演示以及不包括互动的3D模拟器的知识来学习这些属性。由于这些互动物体的普遍性和重要性，该项目有可能加强许多其他学科的努力，例如机器人学或人类辅助技术。与这项研究相结合的是一项计划，通过研究机会和改进的课程材料，在多个层面支持和吸引下一代计算机视觉研究人员。该项目旨在通过四个方向实现这一目标，促进对交互对象的视觉理解。第一个方向旨在建立不受限制的第一人称视频中关节对象的详细3D模型。建立在对发音的物理理解的基础上，第二个方向计划通过关于人如何实现互动以及它可能完成或揭示场景的信息来增强这种物理理解。第三项工作旨在通过在视频帧之间建立3D关联，使系统能够关联并学习正在进行的交互的示例，从而在表达发生之前能够理解它们。第四个方向将对互动物体的理解与产生对整个场景的3D理解的目标联系起来，通过赋予3D重建世界关于可能刚刚离开视线或暂时被遮挡的物体的信念。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in part under the American Rescue Plan Act of 2021 (Public Law 117-2).The human-built world is filled with interactive objects that have parts that can be manipulated by humans, ranging from cabinets with doors to dressers with drawers. In order for intelligent machines to be able to understand and assist humans in realistic settings, they must be able to understand these objects from vision, and especially in unconstrained realistic settings. This understanding must include understanding the interactions as they occur, as well as recognizing the opportunity for interaction (i.e., that a cabinet could be interacted with even when it is untouched). These abilities are beyond the capabilities of current AI systems since these largely deal with interactive objects in restricted settings such as simulation engines. This project aims to build AI systems that can learn these properties by combining knowledge from large-scale first-person-view video demonstrations of interactions by humans as well as from 3D simulators that do not include interaction. The project has the potential to enhance efforts in many other disciplines, for instance robotics or assistive technology for people, due to the ubiquity and importance of these interactive objects. Integrated with the research is a plan to support and engage the next generation of researchers in computer vision at multiple levels via research opportunities and enhanced course materials.This project aims to achieve this goal via four directions that advance the visual understanding of interactive objects. The first direction aims to build detailed 3D models of articulating objects in unconstrained first person-video. Building on this physical understanding of articulation, the second direction plans to enhance this physical understanding with information about how a human would achieve the interaction and what it might accomplish or reveal about the scene. The third effort aims to enable understanding of articulations before they occur by building associations in 3D across frames of a video, letting a system associate and learn from examples of ongoing interactions. The fourth direction connects this understanding of interactive objects with the goal of producing a 3D understanding of the full scene, by endowing 3D reconstructions of the world with beliefs about objects that may be just out of view or temporarily occluded.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.