Individual differences in natural scene semantics investigated with machine learning and neuroimaging

通过机器学习和神经影像研究自然场景语义的个体差异

• 批准号: RGPIN-2021-03127
• 负责人: Charest, Ian
• 金额: $ 2.77万
• 依托单位: Université de Montréal
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=763734
• 关键词: Individual; differences; natural; scene; semantics

The ability to make sense of our visual environment is crucial for adaptive behaviour and survival. Yet, precisely how this subjective, conscious experience of the world emerges in our brain remains poorly understood. The long term aim of my research programme is to provide a new way of thinking about the visual system, placing emphasis on semantic representations and individual differences in the context of natural scene processing. Doing so, we will fill an important gap in our understanding of visual recognition by elucidating the neural underpinnings of semantic processing and its relationship with visual consciousness and subjective experience. How does the brain transform visual information from the retina into high-level semantic representations enabling us to clearly and precisely communicate what we see? Classic accounts suggest functional segregation in two distinct cortical pathways: a "where" and a "what" pathway, a dorsal stream specialised in spatial information ("where") and a ventral stream for category or conceptual information. The ventral stream could therefore be seen as a distributed system where overlapping feature maps encode specific dimensions about particular objects. This view might stem from simple experimental paradigms involving single objects or simplified stimuli, but it is not clear whether this view accurately generalises to naturalistic viewing situations with complex scenes containing multiple objects and concepts. Could there be higher-level (linguistic and semantic conscious form) features, being integrated over object and scene feature maps to provide meaning? To answer these questions, we will combine brain activity measured from electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) while participants view rich and complex natural scenes, with artificial neural networks (ANNs) trained on sentence embeddings, and a battery of innovative behavioural tasks (collected using my online platform for behavioural experiments https://meadows-research.com). The focus for this DG cycle is on moving theories of visual representations into the domains of complex naturalistic stimuli, semantic representations, and individual differences, all of which are relatively unexplored territories in the field. This will be achieved with three core aims: 1- Precisely track where in the brain and when in time after viewing a natural scene, semantic representations are established. 2- Reveal how activity patterns related to scene semantics influence the content of our visual consciousness. 3- Identify links between fine-scale differences in brain representations (i.e. the variations in activity pattern discriminability between individuals) and behaviour (measured in an array of cognitive tasks).

理解我们的视觉环境的能力对于适应性行为和生存至关重要。然而，确切地说，这种主观，有意识的世界在我们大脑中的出现方式仍然很糟糕。我的研究计划的长期目的是提供一种关于视觉系统的思考方式，重点放在语义表示和个体差异中，这在自然场景处理的背景下。这样做，我们将通过阐明语义处理的神经基础及其与视觉意识和主观经验的关系来填补对视觉识别的重要空白。大脑如何将视觉信息从视网膜转化为高级语义表示，使我们能够清楚而精确地传达我们所看到的内容？经典帐户暗示了两个不同的皮质途径的功能隔离：一个“ where”和a“ what d”途径，专门从事空间信息（“ where”）的背面流以及用于类别或概念信息的腹侧流。因此，腹流可以看作是一个分布式系统，其中重叠的特征映射编码有关特定对象的特定维度。这种观点可能源于涉及单个对象或简化刺激的简单实验范式，但是尚不清楚这种观点是否准确地通用了自然主义的观看情况，并具有包含多个对象和概念的复杂场景。是否有更高级别（语言和语义意识形式）功能，在对象和场景图中集成以提供含义？要回答这些问题，我们将结合从电脑学（EEG）和功能性磁共振成像（FMRI）中测量的大脑活动，而参与者则查看富裕而复杂的自然场景，以及对句子嵌入的人工神经网络（ANN），使用我的在线平台进行了我的在线平台进行行为实验，以进行句子嵌入，并使用我的在线行为实验。该DG周期的重点是将视觉表示理论转移到复杂的自然主义刺激，语义表示和个体差异的领域，所有这些差异都是该领域相对未开发的领土。这将通过三个核心目的来实现：1-精确跟踪在大脑中以及观看自然场景后的时间，建立语义表示。 2-揭示与场景语义相关的活动模式如何影响我们的视觉意识的内容。 3-确定大脑表示差异（即活动模式可区分性的变化）与行为（以一系列认知任务来衡量）之间的联系。