Emergence of Abstract Representations in Contextualized Multimodal Models

在人工智能模型和大脑中，抽象表征可以在不同的层次上定义。在较低的层次上，有一些表示是从一些感知变量（如视觉对象的观点）中抽象出来的。更高层次的抽象被定义为独立于输入方式（例如，视觉或听觉）表征意义的表征。除了孤立的概念外，这种表征也可能由感知到的上下文信息形成。例如，声音和视觉对象可能涉及相同的概念，并且感知它的环境可以指导其抽象表示。例如，听到狗叫的声音或看到真正的狗，都与“狗的概念”有关。同样，即使听到有人谈论狗或看到狗屋，也能唤起狗的想法。在这个项目中，我们将专注于更高层次的抽象，独立于输入模式并使用上下文信息。我们将使用深度神经网络（dnn）作为基础，因为这些是最初受视觉皮层启发的分层模型。此外，深度神经网络在一些人工智能应用中是最先进的，例如图像中的对象分类和自然语言处理（NLP）。我们将开发新的多模态深度神经网络，利用不同的共发生输入模式，首先解决计算问题：人工智能模型，特别是基于深度神经网络的模型，如何独立于输入模式学习抽象的语义概念？语境信息的作用是什么？与单模态模型相比，那些学习抽象表示的多模态模型的计算优势（学习期间的数据效率，对输入变化和噪声的鲁棒性）是什么？然后，我们将采用新开发的模型来解释ARENA研究单元的其他项目中收集的人类数据，以了解模型中的抽象表征与人类数据之间的关系，从而使用模型表征大脑表征。总体而言，ARENA研究单元将允许研究人工智能模型和大脑中的抽象表示，并在所有项目之间进行协同紧密合作，以弥合人工智能、神经科学和认知神经科学方面的知识。

Abstract representations, in AI models and in the brain, can be defined at different levels. At a low level there are representations that are abstractions from some perceptual variable (such as the point of view in visual objects). Higher levels of abstraction are defined as representations which characterize meaning independently from input modality (e.g. seeing or hearing). Such representations might also be shaped by contextual information perceived in addition to the isolated concept. For example, a sound and a visual object might refer to the same concept and the context in which it is perceived can guide its abstract representation. For instance, hearing the sound of a barking dog or seeing the actual dog, are both connected to the “dog concept”. Similarly, even hearing someone talking about a dog or seeing a dog house, can evoke the idea of a dog. In this project, we will focus on a higher level of abstraction, independent of input modality and using contextual information. We will employ deep neural networks (DNNs) as a base, since these are hierarchical models originally inspired by the visual cortex. Moreover, DNNs are state-of-the-art for several AI applications, such as object classification in images, and natural language processing (NLP). We will develop new multimodal DNNs leveraging the different co-occurring input modalities, tackling first the computational questions: How do AI models, particularly DNN-based models, learn abstract concepts of semantics, independently of input modality? What is the role of contextual information? What are the computational advantages (data efficiency during learning, robustness to input changes and noise) of those multimodal models that learn abstract representations compared to unimodal models? Then, we will adapt and employ the newly developed models to explain the human data collected in the other projects of the ARENA research unit to understand relations between the abstract representations in the models and human data, and thus, characterize the brain representations using the models. Overall, the ARENA research unit will allow the study of abstract representations in AI models and brain with a synergistically tight collaboration among all the projects to bridge knowledge in AI, neuroscience and cognitive neuroscience.