The Hierarchical Structure of Abstract Scene Representations

高效的人类思维的一个关键特征是能够从特定的以前的经验中抽象出来，并建立抽象的表征，使其能够成功地应用于我们不断变化的环境。也就是说，就像我们能够理解以前从未听过的句子一样，因为我们知道单词的含义以及它们需要如何排列以形成意义的规则，我们可以通过访问对象的身份并了解控制其空间排列的规则来轻松理解新环境（场景语法；Võ， 2021）。根据ARENA的总体目标，这个项目的目的是理解现实世界场景中物体的抽象表示的出现和组成，从而实现有效的行为。我们打算通过使用不同的输入方式（例如图像与单词）和不同的任务（例如记忆与搜索）系统地探索场景语法中层次结构的存在来实现这一目标。一个关键的假设涉及场景中存在的子集群——“短语”——它们同样由一个较大的对象——一个“锚定对象”——组成，它反过来预测其他较小对象的身份和位置。为了与我们的环境进行交互，重要的是建立与外部世界结构平行的内部表示，例如，通过建立允许更有效的对象识别或定位的预测。由于ARENA的多学科组成，我们将能够通过与计算机科学的同事合作来验证我们的假设。例如，我们将比较对象和场景呈现的行为和神经反应，以及早期和晚期深度神经网络的输出。我们的综合方法将包括三个核心研究线，以确保对以下方面进行彻底、有原则的调查：1)由物体图像或单词标签引出的抽象场景表示的结构；2)使用EEG和fMRI从大脑反应中估计的现有搜索模板的内容和结构；3)在人工3D VR环境中交互过程中生成和修改新的抽象对象表示。成功地解决所提出的目标可能会对我们如何在计算机科学和心理学中研究复杂的、自然的环境中的认知过程产生变革性的影响，下一步可能会为开发新的、更有效的场景理解计算模型提供信息。

A key feature of the efficient human mind is the ability to abstract away from specific previous experiences and build abstract representations that allow for their successful application to our constantly changing environment. That is, just like we are able to understand sentences we have never heard before because we know the meaning of the words and the rules of how they need to be arranged to form meaning, we can easily understand new environments by accessing the identities of objects and knowing the rules that govern their spatial arrangements (scene grammar; Võ, 2021). Along the lines of ARENA’s overarching objectives, the aim of this project is therefore to understand the emergence and composition of abstract representations of objects in real-world scenes which allow for efficient behavior. We intend to do pursue this aim by systematically probing the existence of a hierarchical structure in scene grammar using different input modalities (e.g. images vs. words) and different tasks (e.g. memorization vs. search). A key assumption concerns the existence of subclusters within scenes – “phrases” – that again consist of a larger object ¬– an “anchor object” – which in turn predicts the identity and location of other, smaller objects. In order to interact with our environment, it is important to establish internal representations that parallel the structure of the external world, e.g. by setting up predictions that allow for more efficient object identification or localization. Thanks to the multi-disciplinary composition of ARENA, we will be able to test our hypothesis by joining forces with our colleagues from computer science. For instance, we will compare behavioral and neural responses to the presentations of objects and scenes with outputs of early versus late layers of deep neural networks. Our comprehensive approach will include three core lines of research to ensure a thorough, principled investigation of I) the structure of abstract scene representations elicited by either images of objects or word labels, II) the content and structure of existing search templates estimated from brain responses using EEG and fMRI, and III) the generation and modification of new abstract object representations over the course of interactions in artificial 3D VR environments. Successfully tackling the proposed aims could have transformational effects on how we study cognitive processes in complex, naturalistic settings within computer science and psychology, and in a next step could inform the development of new, more efficient computational models of scene understanding.