Physion: Evaluating Physical Prediction from Vision in Humans and Machines-猫眼课题宝

DOI：

发表时间：

2021-06

期刊：

ArXiv

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Daniel Bear;E. Wang;Damian Mrowca;Felix Binder;Hsiau-Yu Fish Tung;R. Pramod;Cameron Holdaway;Sirui Tao;Kevin A. Smith;Li Fei-Fei-Li-Fei-Fei-48004138;N. Kanwisher;J. Tenenbaum;Daniel Yamins;Judith E. Fan

虽然目前的视觉算法在许多具有挑战性的任务中表现出色，但尚不清楚它们对现实世界环境的物理动力学的理解程度。在这里，我们介绍了Physion，这是一个数据集和基准，用于严格评估预测物理场景如何随时间演变的能力。我们的数据集具有广泛的物理现象的逼真模拟，包括刚体和软体碰撞，稳定的多对象配置，滚动，滑动和抛射运动，从而提供比以前的基准更全面的挑战。我们使用Physion对一套在架构、学习目标、输入输出结构和训练数据方面各不相同的模型进行了基准测试。与此同时，我们在同一组场景中获得了人类预测行为的精确测量值，使我们能够直接评估任何模型对人类行为的近似程度。我们发现，学习以对象为中心的表示的视觉算法通常优于那些不学习的算法，但仍然远远低于人类的表现。另一方面，直接访问物理状态信息的图形神经网络表现得更好，并且做出的预测与人类更相似。这些结果表明，提取场景的物理表示是视觉算法中实现人类水平和类人物理理解的主要瓶颈。我们已经公开发布了所有数据和代码，以便于使用Physion以完全可复制的方式对其他模型进行基准测试，从而能够系统地评估视觉算法的进展，这些算法可以像人类一样强大地理解物理环境。

While current vision algorithms excel at many challenging tasks, it is unclear how well they understand the physical dynamics of real-world environments. Here we introduce Physion, a dataset and benchmark for rigorously evaluating the ability to predict how physical scenarios will evolve over time. Our dataset features realistic simulations of a wide range of physical phenomena, including rigid and soft-body collisions, stable multi-object configurations, rolling, sliding, and projectile motion, thus providing a more comprehensive challenge than previous benchmarks. We used Physion to benchmark a suite of models varying in their architecture, learning objective, input-output structure, and training data. In parallel, we obtained precise measurements of human prediction behavior on the same set of scenarios, allowing us to directly evaluate how well any model could approximate human behavior. We found that vision algorithms that learn object-centric representations generally outperform those that do not, yet still fall far short of human performance. On the other hand, graph neural networks with direct access to physical state information both perform substantially better and make predictions that are more similar to those made by humans. These results suggest that extracting physical representations of scenes is the main bottleneck to achieving human-level and human-like physical understanding in vision algorithms. We have publicly released all data and code to facilitate the use of Physion to benchmark additional models in a fully reproducible manner, enabling systematic evaluation of progress towards vision algorithms that understand physical environments as robustly as people do.

Physion: Evaluating Physical Prediction from Vision in Humans and Machines

文献摘要

被引文献

相似文献