Emergence of Abstract Representations in Contextualized Multimodal Models

AI模型和大脑中的抽象表示可以在不同的层次上定义。在低层次上，有一些表征是从一些感知变量（如视觉对象的观点）中抽象出来的。更高层次的抽象被定义为表征独立于输入模态（例如，看到或听到）的含义的表示。除了孤立的概念之外，这种表征还可能受到感知的上下文信息的影响。例如，一个声音和一个视觉对象可能指的是同一个概念，而感知它的上下文可以指导它的抽象表示。例如，听到狗叫的声音或看到真正的狗，都与“狗的概念”有关。同样，即使听到有人谈论狗或看到狗屋，也会唤起狗的想法。在这个项目中，我们将专注于更高层次的抽象，独立于输入模态和使用上下文信息。我们将使用深度神经网络（DNN）作为基础，因为这些是最初受视觉皮层启发的分层模型。此外，DNN是几个AI应用的最新技术，例如图像中的对象分类和自然语言处理（NLP）。我们将利用不同的共现输入模态开发新的多模态DNN，首先解决计算问题：AI模型，特别是基于DNN的模型，如何独立于输入模态学习语义的抽象概念？上下文信息的作用是什么？与单峰模型相比，学习抽象表示的多模态模型的计算优势是什么（学习过程中的数据效率，对输入变化和噪声的鲁棒性）？然后，我们将采用新开发的模型来解释在竞技场研究单位的其他项目中收集的人类数据，以了解模型中的抽象表征与人类数据之间的关系，从而使用模型来表征大脑表征。总体而言，竞技场研究部门将允许研究AI模型和大脑中的抽象表示，并在所有项目之间进行协同紧密合作，以弥合AI，神经科学和认知神经科学的知识。

Abstract representations, in AI models and in the brain, can be defined at different levels. At a low level there are representations that are abstractions from some perceptual variable (such as the point of view in visual objects). Higher levels of abstraction are defined as representations which characterize meaning independently from input modality (e.g. seeing or hearing). Such representations might also be shaped by contextual information perceived in addition to the isolated concept. For example, a sound and a visual object might refer to the same concept and the context in which it is perceived can guide its abstract representation. For instance, hearing the sound of a barking dog or seeing the actual dog, are both connected to the “dog concept”. Similarly, even hearing someone talking about a dog or seeing a dog house, can evoke the idea of a dog. In this project, we will focus on a higher level of abstraction, independent of input modality and using contextual information. We will employ deep neural networks (DNNs) as a base, since these are hierarchical models originally inspired by the visual cortex. Moreover, DNNs are state-of-the-art for several AI applications, such as object classification in images, and natural language processing (NLP). We will develop new multimodal DNNs leveraging the different co-occurring input modalities, tackling first the computational questions: How do AI models, particularly DNN-based models, learn abstract concepts of semantics, independently of input modality? What is the role of contextual information? What are the computational advantages (data efficiency during learning, robustness to input changes and noise) of those multimodal models that learn abstract representations compared to unimodal models? Then, we will adapt and employ the newly developed models to explain the human data collected in the other projects of the ARENA research unit to understand relations between the abstract representations in the models and human data, and thus, characterize the brain representations using the models. Overall, the ARENA research unit will allow the study of abstract representations in AI models and brain with a synergistically tight collaboration among all the projects to bridge knowledge in AI, neuroscience and cognitive neuroscience.