权益分类	功能权益	普通用户	{{item.name}}会员
{{category.name}}	{{benefitItem.name}}

CAREER: Toward Video2Sim: Turning Real World Videos into Simulations

职业：走向Video2Sim：将现实世界的视频变成模拟

基本信息

批准号：
1942981
负责人：
Jia Deng
金额：
$ 55万
依托单位：
Princeton University
依托单位国家：
美国
项目类别：
Continuing Grant
财政年份：
2020
资助国家：
美国
起止时间：
2020-04-01 至 2025-03-31
项目状态：
未结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1942981&HistoricalAwards=false
关键词：
CAREER Toward Video2Sim Turning Real

项目摘要

This project develops new technology toward Video2Sim: automatically converting a video into a virtual world, where scenes are reconstructed, actions are re-enacted, and alternative outcomes are simulated by a computer. Such a system does not yet exist due to the limitations of existing technology, and as a result, virtual worlds need to be manually and laboriously constructed. Video2Sim is useful because virtual worlds can be used to train and evaluate AI systems. For example, videos of traffic accidents can be converted into simulations to test autonomous cars, or videos of kitchen scenes to test home robots. Simulation is more scalable and cost-effective than real world experiments and is particularly suited for machine learning algorithms that require a lot of training data. Furthermore, such an automated system can leverage a large number of videos to provide a comprehensive coverage of rare events, which is essential for evaluating and assuring the safety of autonomous systems. Therefore, Video2Sim has the potential to benefit a broad range of applications including robotics, healthcare, and transportation. Research in this project is integrated with K12, undergraduate, and graduate education through research training, course development and outreach events. This research develops key techniques toward a Video2Sim system with a focus on 3D shape and motion. This effort is organized into two thrusts: (1) reconstructing 3D shape and motion and (2) simulating dynamics and behavior. The goal of thrust 1 is to recover 3D shape and motion of a full scene from a monocular video, such that we can re-render the scene and re-enact the events from an arbitrary view. The focus is on developing methods to recover detailed 3D shape and 3D motion from arbitrary unconstrained videos. The goal of thrust 2 is to recover the underlying dynamics of a scene, such that we can not only re-enact the actual events but also simulate alternative outcomes. The focus is on developing methods to infer not only physical properties of passive objectives but also behavior models of agents, that is, entities that do not just move passively according to external forces but can plan and initiate their own actions.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目开发了面向Video2Sim的新技术：自动将视频转换为虚拟世界，在虚拟世界中重建场景，重新制定行动，并通过计算机模拟替代结果。由于现有技术的限制，这样的系统还不存在，因此，虚拟世界需要手动和费力地构建。Video2Sim很有用，因为虚拟世界可以用来训练和评估AI系统。例如，交通事故的视频可以转换成模拟来测试自动驾驶汽车，或者厨房场景的视频来测试家用机器人。仿真比真实的世界实验更具可扩展性和成本效益，特别适合需要大量训练数据的机器学习算法。此外，这种自动化系统可以利用大量视频来提供对罕见事件的全面覆盖，这对于评估和确保自主系统的安全性至关重要。因此，Video2Sim有可能使广泛的应用受益，包括机器人，医疗保健和运输。该项目的研究通过研究培训，课程开发和推广活动与K12，本科和研究生教育相结合。本研究开发了一个Video2Sim系统的关键技术，重点是3D形状和运动。这项工作分为两个重点：（1）重建3D形状和运动以及（2）模拟动力学和行为。推力1的目标是从单目视频中恢复完整场景的3D形状和运动，这样我们就可以从任意视图重新渲染场景并重新制定事件。重点是开发从任意无约束视频中恢复详细3D形状和3D运动的方法。推力2的目标是恢复场景的潜在动力学，这样我们不仅可以重现实际事件，还可以模拟替代结果。该奖项的重点是开发方法，不仅推断被动目标的物理特性，还推断代理人的行为模型，即不只是被动地根据外力移动，而是可以计划和启动自己的行动的实体。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

期刊论文数量（9）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

Multiview Stereo with Cascaded Epipolar RAFT

DOI：
10.48550/arxiv.2205.04502
发表时间：
2022-05
期刊：
ArXiv
影响因子：
0
作者：
Zeyu Ma;Zachary Teed;Jia Deng
通讯作者：
Zeyu Ma;Zachary Teed;Jia Deng

Infinite Photorealistic Worlds Using Procedural Generation

DOI：
10.1109/cvpr52729.2023.01215
发表时间：
2023-06
期刊：
2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
影响因子：
0
作者：
Alexander R. E. Raistrick;Lahav Lipson;Zeyu Ma;Lingjie Mei;Mingzhe Wang;Yiming Zuo;Karhan Kayan;Hongyu Wen;Beining Han;Yihan Wang;Alejandro Newell;Hei Law;Ankit Goyal;Kaiyu Yang;Jia Deng
通讯作者：
Alexander R. E. Raistrick;Lahav Lipson;Zeyu Ma;Lingjie Mei;Mingzhe Wang;Yiming Zuo;Karhan Kayan;Hongyu Wen;Beining Han;Yihan Wang;Alejandro Newell;Hei Law;Ankit Goyal;Kaiyu Yang;Jia Deng

Tangent Space Backpropagation for 3D Transformation Groups