CAREER: Digitize and Simulate the Large Physical World via Knowledge-Grounded Scene Representation

许多领域，如农业机器人，都需要人类和智能代理对物理世界有共同的理解，才能做出明智的决定。这就要求智能机器，能够有效地帮助人类，了解物理世界，模拟各种可能性，并预测和评估结果。目前的计算机视觉系统主要侧重于理解和模拟观察到的现象，缺乏这些能力。同样，特定于领域的模拟器无法实现这一目标，因为它们缺乏现实世界的基础。该项目旨在构建人工智能系统，能够创建大型3D世界的数字副本，并忠实地模拟各种反现实场景，从而使用户能够评估不同决策和行动的结果。这项工作的核心是一种可操作的、基于知识的场景表示，有助于现实世界的建模和模拟。该项目有可能扩大各学科的进展，如机器人和农业。该项目旨在通过推进数字化和模拟物理世界的四个方向来实现这一目标。为了实现这一目标，该项目提供了一个框架，它将(I)从真实世界的视频中创建可操作的场景表示；(Ii)从视觉观察推断数字世界的物理参数；(Iii)对数字孪生兄弟进行基于物理的模拟，从而能够创建真实和准确的反事实；以及(Iv)将数字孪生兄弟与领域知识和各种应用的建模相结合。在研究研究的基础上，该项目将开发一个数字双胞胎教育平台，可应用于本科生和研究生教育、指导和K-12推广活动，以吸引下一代研究人员参与计算。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Numerous fields, such as agricultural robotics, necessitate that humans and intelligent agents have a shared understanding of the physical world to be able to make informed decisions. This requires that intelligent machines, to effectively assist humans, understand the physical world, simulate various possibilities, and predict and evaluate outcomes. Current computer vision systems, which largely focus on understanding and modeling observed phenomena, fall short of these capabilities. Similarly, domain-specific simulators fail to achieve this goal as they lack real-world grounding. This project aims to construct AI systems capable of creating a digital replica of the large 3D world and faithfully simulating various counterfactual scenarios, thereby enabling users to assess the outcomes of different decisions and actions. Central to this work is an actionable, knowledge-grounded scene representation, facilitating real-world modeling and simulation. The project has the potential to amplify advancements in various disciplines, such as robotics and agriculture. This project aims to achieve this goal via four directions that advance digitizing and simulating the physical world. To achieve this goal, the project offers a framework that will (i) create an actionable scene representation from real-world videos; (ii) infer physical parameters of the digital world from visual observations; (iii) perform physics-grounded simulation over the digital twin that enables the creation of realistic and accurate counterfactuals; and (iv) integrate the digital twin with domain knowledge and modeling for various applications. Based on the research studies, the project will develop a digital twin education platform that can be applied in undergraduate and graduate education, mentoring, and K-12 outreach activities to engage the next generation of researchers in computing.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.