Individual differences in natural scene semantics investigated with machine learning and neuroimaging

通过机器学习和神经影像研究自然场景语义的个体差异

• 批准号: RGPIN-2021-03127
• 负责人: Charest, Ian
• 金额: $ 2.77万
• 依托单位: Université de Montréal
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=763734
• 关键词: Individual; differences; natural; scene; semantics

The ability to make sense of our visual environment is crucial for adaptive behaviour and survival. Yet, precisely how this subjective, conscious experience of the world emerges in our brain remains poorly understood. The long term aim of my research programme is to provide a new way of thinking about the visual system, placing emphasis on semantic representations and individual differences in the context of natural scene processing. Doing so, we will fill an important gap in our understanding of visual recognition by elucidating the neural underpinnings of semantic processing and its relationship with visual consciousness and subjective experience. How does the brain transform visual information from the retina into high-level semantic representations enabling us to clearly and precisely communicate what we see? Classic accounts suggest functional segregation in two distinct cortical pathways: a "where" and a "what" pathway, a dorsal stream specialised in spatial information ("where") and a ventral stream for category or conceptual information. The ventral stream could therefore be seen as a distributed system where overlapping feature maps encode specific dimensions about particular objects. This view might stem from simple experimental paradigms involving single objects or simplified stimuli, but it is not clear whether this view accurately generalises to naturalistic viewing situations with complex scenes containing multiple objects and concepts. Could there be higher-level (linguistic and semantic conscious form) features, being integrated over object and scene feature maps to provide meaning? To answer these questions, we will combine brain activity measured from electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) while participants view rich and complex natural scenes, with artificial neural networks (ANNs) trained on sentence embeddings, and a battery of innovative behavioural tasks (collected using my online platform for behavioural experiments https://meadows-research.com). The focus for this DG cycle is on moving theories of visual representations into the domains of complex naturalistic stimuli, semantic representations, and individual differences, all of which are relatively unexplored territories in the field. This will be achieved with three core aims: 1- Precisely track where in the brain and when in time after viewing a natural scene, semantic representations are established. 2- Reveal how activity patterns related to scene semantics influence the content of our visual consciousness. 3- Identify links between fine-scale differences in brain representations (i.e. the variations in activity pattern discriminability between individuals) and behaviour (measured in an array of cognitive tasks).

理解视觉环境的能力对于适应行为和生存至关重要。然而，这种主观的、有意识的世界体验是如何在我们的大脑中出现的，人们仍然知之甚少。我的研究计划的长期目标是为视觉系统提供一种新的思维方式，强调自然场景处理背景下的语义表征和个体差异。这样做，我们将通过阐明语义加工的神经基础及其与视觉意识和主观体验的关系来填补我们对视觉识别理解的一个重要空白。大脑如何将视网膜中的视觉信息转化为高级语义表示，使我们能够清楚而准确地传达我们所看到的？经典的描述表明，在两条不同的皮质通路中存在功能分离：一条“哪里”和一条“什么”通路，一条专门处理空间信息的背侧流(“哪里”)和一条腹侧流，用于类别或概念信息。因此，腹侧流可以被视为一个分布式系统，其中重叠的特征映射编码关于特定对象的特定维度。这一观点可能源于涉及单一对象或简化刺激的简单实验范式，但目前尚不清楚这种观点是否准确地概括为具有包含多个对象和概念的复杂场景的自然主义观看情景。有没有可能有更高层次的(语言和语义意识形式)特征，通过整合在对象和场景特征地图上来提供意义？为了回答这些问题，我们将在参与者观看丰富而复杂的自然场景时，结合从脑电(EEG)和功能磁共振成像(FMRI)测量的大脑活动，使用人工神经网络(ANN)训练句子嵌入，以及一系列创新的行为任务(使用我的行为实验在线平台https://meadows-research.com).收集这个DG循环的重点是将视觉表征理论转移到复杂的自然主义刺激、语义表征和个体差异领域，所有这些都是该领域相对未被探索的领域。这将通过三个核心目标来实现：1-在观看自然场景后，准确跟踪大脑中的位置和时间，建立语义表征。2-揭示与场景语义相关的活动模式如何影响我们视觉意识的内容。3-确定大脑表征的细微差异(即个体之间活动模式辨别能力的差异)与行为(通过一系列认知任务衡量)之间的联系。