CAREER: Toward Video2Sim: Turning Real World Videos into Simulations

该项目为Video2Sim开发了新技术：自动将视频转换为虚拟世界，在虚拟世界中重构场景，重新制定动作，并由计算机模拟不同的结果。由于现有技术的限制，目前还不存在这样的系统，因此虚拟世界需要人工费力地构建。Video2Sim很有用，因为虚拟世界可以用来训练和评估人工智能系统。例如，交通事故的视频可以转换成模拟来测试自动驾驶汽车，或者厨房场景的视频来测试家用机器人。模拟比现实世界的实验更具可扩展性和成本效益，特别适合需要大量训练数据的机器学习算法。此外，这样的自动化系统可以利用大量的视频来提供罕见事件的全面覆盖，这对于评估和确保自主系统的安全性至关重要。因此，Video2Sim有潜力使广泛的应用受益，包括机器人、医疗保健和运输。该项目的研究通过研究培训、课程开发和外展活动与K12、本科和研究生教育相结合。本研究开发了Video2Sim系统的关键技术，重点是三维形状和运动。这项工作分为两个重点：(1)重建3D形状和运动；(2)模拟动力学和行为。推力1的目标是从单目视频中恢复完整场景的3D形状和运动，这样我们就可以从任意视图重新渲染场景和重新制定事件。重点是开发方法，以恢复详细的3D形状和3D运动从任意无约束的视频。推力2的目标是恢复场景的潜在动态，这样我们不仅可以重新制定实际事件，还可以模拟其他结果。重点是开发方法，不仅可以推断被动目标的物理特性，还可以推断代理的行为模型，即不只是被动地根据外力移动的实体，还可以计划和启动自己的行动。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project develops new technology toward Video2Sim: automatically converting a video into a virtual world, where scenes are reconstructed, actions are re-enacted, and alternative outcomes are simulated by a computer. Such a system does not yet exist due to the limitations of existing technology, and as a result, virtual worlds need to be manually and laboriously constructed. Video2Sim is useful because virtual worlds can be used to train and evaluate AI systems. For example, videos of traffic accidents can be converted into simulations to test autonomous cars, or videos of kitchen scenes to test home robots. Simulation is more scalable and cost-effective than real world experiments and is particularly suited for machine learning algorithms that require a lot of training data. Furthermore, such an automated system can leverage a large number of videos to provide a comprehensive coverage of rare events, which is essential for evaluating and assuring the safety of autonomous systems. Therefore, Video2Sim has the potential to benefit a broad range of applications including robotics, healthcare, and transportation. Research in this project is integrated with K12, undergraduate, and graduate education through research training, course development and outreach events. This research develops key techniques toward a Video2Sim system with a focus on 3D shape and motion. This effort is organized into two thrusts: (1) reconstructing 3D shape and motion and (2) simulating dynamics and behavior. The goal of thrust 1 is to recover 3D shape and motion of a full scene from a monocular video, such that we can re-render the scene and re-enact the events from an arbitrary view. The focus is on developing methods to recover detailed 3D shape and 3D motion from arbitrary unconstrained videos. The goal of thrust 2 is to recover the underlying dynamics of a scene, such that we can not only re-enact the actual events but also simulate alternative outcomes. The focus is on developing methods to infer not only physical properties of passive objectives but also behavior models of agents, that is, entities that do not just move passively according to external forces but can plan and initiate their own actions.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.