Collaborative Research: NCS-FR: Beyond the ventral stream: Reverse engineering the neurocomputational basis of physical scene understanding in the primate brain

过去十年见证了人工智能的惊人革命，深度神经网络在识别图像中的物体和音频记录中的单词等问题上的表现突然接近人类水平。尽管这些成就令人印象深刻，但它们远不及人类的智慧。当前人工智能与人类智能之间的关键差距在于，除了对输入模式进行分类之外，人类还建立了对世界的心理模型。这个项目从物理场景理解的问题开始：一个人如何提取不仅仅是视觉世界中物体的身份和位置，还有这些物体的物理特性，它们的位置和速度，它们彼此之间的关系，作用在它们身上的力，以及可能施加在它们身上的力的影响。有一种假设是，人类用物理世界的结构化心理模型来表示这些信息，并使用该模型来预测接下来会发生什么，就像电子游戏中的物理引擎生成虚拟世界的物理可信的未来状态一样。为了验证这一想法，将建立物理场景理解的计算模型，并测试它们在各种场景中预测物理世界未来状态的能力。然后将这些模型的性能与人类和更传统的深度网络模型进行比较，无论是在每个任务的准确性方面，还是在错误模式方面。结合物理世界的结构化表征的计算模型将在标准卷积神经网络的基础上进行测试，以检验它们解释人脑（使用功能磁共振成像）和猴子大脑（使用直接神经记录）的神经反应的能力。这些计算模型将为物理场景理解如何在人类大脑中工作提供第一个明确的理论，同时提高人工智能系统解决相同问题的能力。因为理解和预测物理世界的能力对于计划任何行动都是至关重要的，这项工作有望帮助推进许多需要这种计划的技术，从机器人到自动驾驶汽车再到脑机接口。每个参与的实验室还将扩大其在本科、研究生和博士后阶段招聘、培训和指导女性和少数族裔的现有记录。最后，合作实验室将通过公开讲座、网站和外联活动，继续并增加它们对向公众传播科学的参与。深度神经网络已经彻底改变了计算机中的物体识别以及灵长类动物大脑中对物体识别的理解，但物体识别只是视觉的一个方面，腹侧神经流只是许多大脑系统中的一个。研究物理场景理解是将这种逆向工程方法扩展到心智和大脑的其他部分的一步。预测接下来会发生什么并计划有效的行动需要理解视觉世界中的物理基础和物理关系。然而，人类如何做到这一点，机器如何做到这一点，我们不得而知。在这个项目中，这两个挑战都是通过构建图像可计算的、神经可映射的物理场景理解和预测计算模型（线程I）来解决的，并使用这些模型作为大脑如何完成这些任务的明确假设，然后将用人类（线程II）和非人类灵长类动物（线程III）的行为和神经数据进行测试。这个项目的目标是在理解上实现革命性的飞跃：从小规模的特殊情况模型和孤立的实验测试，到灵长类动物大脑主要区域的综合大规模、通用模型，从功能上解释我们面对的每个场景中感知经验的大部分直接内容。这项工作将通过开发第一个能够理解和预测人类物理场景的图像可计算模型来推进理论。除了对思想和大脑的理解之外，这项研究还与人工智能和机器人技术（需要对物理场景的理解）以及脑机接口（需要对相关神经代码的理解）直接相关。对于更广泛的研究界，该项目将a)开发公共数据集、基准任务和挑战，b)主持对抗性合作以应对这些挑战，以及c)举办跨学科研讨会，将心理学、人工智能和神经科学等研究团体联系起来，以解决跨越这些领域的基本问题。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The last ten years have witnessed an astonishing revolution in AI, with deep neural networks suddenly approaching human-level performance on problems like recognizing objects in an image and words in an audio recording. But impressive as these feats are, they fall far short of human-like intelligence. The critical gap between current AI and human intelligence is that, beyond just classifying patterns of input, humans build mental models of the world. This project begins with the problem of physical scene understanding: how one extracts not just the identities and locations of objects in the visual world, but also the physical properties of those objects, their positions and velocities, their relationships to each other, the forces acting upon them, and the effects of forces that could be exerted on them. It is hypothesized that humans represent this information in a structured mental model of the physical world, and use that model to predict what will happen next, much as the physics engine in a video game generates physically plausible future states of virtual worlds. To test this idea, computational models of physical scene understanding will be built and tested for their ability to predict future states of the physical world in a variety of scenarios. Performance of these models will then be compared to humans and to more traditional deep network models, both in terms of their accuracy on each task, and their patterns of errors. Computational models that incorporate structured representations of the physical world will then be tested against standard convolutional neural networks in their ability to explain neural responses of the human brain (using fMRI) and the monkey brain (using direct neural recording). These computational models will provide the first explicit theories of how physical scene understanding might work in the human brain, at the same time advancing the ability of AI systems to solve the same problems. Because the ability to understand and predict the physical world is essential for planning any action, this work is expected to help advance many technologies that require such planning, from robotics to self-driving cars to brain-machine interfaces. Each of the participating labs will also expand their established track records of recruiting, training, and mentoring women and under-represented minorities at the undergraduate, graduate, and postdoctoral levels. Finally, the collaborating laboratories will continue and increase their involvement in the dissemination of science to the general public, via public talks, web sites, and outreach activities.Deep neural networks have revolutionized object recognition in computers as well as understanding of object recognition in the primate brain, but object recognition is just one aspect of vision, and the ventral stream is just one of many brain systems. Studying physical scene understanding is a step toward scaling this reverse-engineering approach up to the rest of the mind and brain. Predicting what will happen next and planning effective action requires understanding the physical basis and physical relationships in the visual world. Yet it is unknown how humans do this or how machines could. Both challenges are addressed in this project by the building of image computable, neurally mappable computational models of physical scene understanding and prediction (Thread I), and using these models as explicit hypotheses for how the brain might accomplish these tasks, which will then be tested with behavioral and neural data from humans (Thread II) and non-human primates (Thread III). This project aims to make a transformative leap in understanding: from small-scale, special-case models and isolated experimental tests to an integrated large-scale, general-purpose model of a major swathe of the primate brain, that functionally explains much of the immediate content of our perceptual experience in every scene that confronts us. The work will advance theory by developing the first image-computable models capable of human-level physical scene understanding and prediction. Beyond understanding of the mind and brain, this research is directly relevant to AI and robotics (which require physical scene understanding), and brain-machine interfaces (which require understanding of the relevant neural codes). For the broader research community, the project will a) develop public datasets, benchmark tasks, and challenges, b) host adversarial collaborations to address these challenges, and c) host interdisciplinary workshops linking research communities from psychology to AI to neuroscience to address the fundamental questions that span these fields.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.