ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
复制标题
DOI：
--
复制
发表时间：
2020-07
期刊：
影响因子：
--
通讯作者：
Chuang Gan;Jeremy Schwartz;S. Alter;Martin Schrimpf;James Traer;Julian De Freitas;J. Kubilius;Abhishek Bhandwaldar;Nick Haber;Megumi Sano;Kuno Kim;E. Wang;Damian Mrowca;Michael Lingelbach;Aidan Curtis;Kevin T. Feigelis;Daniel Bear;Dan Gutfreund;David Cox;J. DiCarlo;Josh H. McDermott;J. Tenenbaum;Daniel L. K. Yamins
中科院分区：
文献类型：
--
作者：
Chuang Gan;Jeremy Schwartz;S. Alter;Martin Schrimpf;James Traer;Julian De Freitas;J. Kubilius;Abhishek Bhandwaldar;Nick Haber;Megumi Sano;Kuno Kim;E. Wang;Damian Mrowca;Michael Lingelbach;Aidan Curtis;Kevin T. Feigelis;Daniel Bear;Dan Gutfreund;David Cox;J. DiCarlo;Josh H. McDermott;J. Tenenbaum;Daniel L. K. Yamins
我们介绍了三个世界（TDW），交互式多模态物理仿真平台。使用TDW，用户可以在各种丰富的3D环境中模拟移动的代理和对象之间的高保真传感数据和物理交互。TDW有几个独特的属性：1）实时近照片逼真的图像渲染质量; 2）具有高质量渲染材料的对象和环境库，以及使用户能够自定义资产库的例程; 3）用于有效构建新环境类的生成过程4）高保真音频渲染; 5）用于各种材料类型的可信和现实的物理交互，包括布、液体和可变形物体; 6）用作AI代理的实施例的一系列“化身”类型，具有用户化身定制的选项;以及7）支持人类与VR设备的交互。TDW还提供了丰富的API，使多个代理能够在仿真中进行交互，并返回一系列代表世界状态的传感器和物理数据。我们围绕计算机视觉、机器学习和认知科学的新兴研究方向，提出了由该平台支持的初步实验，包括多模态物理场景理解、多智能体交互、“像孩子一样学习”的模型以及人类和神经网络的注意力研究。模拟平台将向公众开放。
We introduce ThreeDWorld (TDW), a platform for interactive multi-modal physical simulation. With TDW, users can simulate high-fidelity sensory data and physical interactions between mobile agents and objects in a wide variety of rich 3D environments. TDW has several unique properties: 1) realtime near photo-realistic image rendering quality; 2) a library of objects and environments with materials for high-quality rendering, and routines enabling user customization of the asset library; 3) generative procedures for efficiently building classes of new environments 4) high-fidelity audio rendering; 5) believable and realistic physical interactions for a wide variety of material types, including cloths, liquid, and deformable objects; 6) a range of "avatar" types that serve as embodiments of AI agents, with the option for user avatar customization; and 7) support for human interactions with VR devices. TDW also provides a rich API enabling multiple agents to interact within a simulation and return a range of sensor and physics data representing the state of the world. We present initial experiments enabled by the platform around emerging research directions in computer vision, machine learning, and cognitive science, including multi-modal physical scene understanding, multi-agent interactions, models that "learn like a child", and attention studies in humans and neural networks. The simulation platform will be made publicly available.