Individual differences in natural scene semantics investigated with machine learning and neuroimaging

通过机器学习和神经影像研究自然场景语义的个体差异

• 批准号: RGPIN-2021-03127
• 负责人: Charest, Ian
• 金额: $ 2.77万
• 依托单位: Université de Montréal
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=763734
• 关键词: Individual; differences; natural; scene; semantics

The ability to make sense of our visual environment is crucial for adaptive behaviour and survival. Yet, precisely how this subjective, conscious experience of the world emerges in our brain remains poorly understood. The long term aim of my research programme is to provide a new way of thinking about the visual system, placing emphasis on semantic representations and individual differences in the context of natural scene processing. Doing so, we will fill an important gap in our understanding of visual recognition by elucidating the neural underpinnings of semantic processing and its relationship with visual consciousness and subjective experience. How does the brain transform visual information from the retina into high-level semantic representations enabling us to clearly and precisely communicate what we see? Classic accounts suggest functional segregation in two distinct cortical pathways: a "where" and a "what" pathway, a dorsal stream specialised in spatial information ("where") and a ventral stream for category or conceptual information. The ventral stream could therefore be seen as a distributed system where overlapping feature maps encode specific dimensions about particular objects. This view might stem from simple experimental paradigms involving single objects or simplified stimuli, but it is not clear whether this view accurately generalises to naturalistic viewing situations with complex scenes containing multiple objects and concepts. Could there be higher-level (linguistic and semantic conscious form) features, being integrated over object and scene feature maps to provide meaning? To answer these questions, we will combine brain activity measured from electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) while participants view rich and complex natural scenes, with artificial neural networks (ANNs) trained on sentence embeddings, and a battery of innovative behavioural tasks (collected using my online platform for behavioural experiments https://meadows-research.com). The focus for this DG cycle is on moving theories of visual representations into the domains of complex naturalistic stimuli, semantic representations, and individual differences, all of which are relatively unexplored territories in the field. This will be achieved with three core aims: 1- Precisely track where in the brain and when in time after viewing a natural scene, semantic representations are established. 2- Reveal how activity patterns related to scene semantics influence the content of our visual consciousness. 3- Identify links between fine-scale differences in brain representations (i.e. the variations in activity pattern discriminability between individuals) and behaviour (measured in an array of cognitive tasks).

理解我们视觉环境的能力对于适应行为和生存至关重要。然而，这种主观的、有意识的世界体验是如何在我们的大脑中出现的，我们仍然知之甚少。我的研究计划的长期目标是提供一种新的思维方式的视觉系统，把重点放在语义表征和自然场景处理的背景下的个体差异。这样做，我们将通过阐明语义处理的神经基础及其与视觉意识和主观经验的关系，填补我们对视觉识别理解的重要空白。大脑如何将来自视网膜的视觉信息转化为高级语义表征，使我们能够清晰准确地传达我们所看到的内容？经典的解释表明，功能分离在两个不同的皮层通路：一个“在哪里”和“什么”的途径，背流专门用于空间信息（“在哪里”）和腹流的类别或概念信息。因此，腹侧流可以被看作是一个分布式系统，其中重叠的特征图编码了关于特定对象的特定维度。这种观点可能源于涉及单一物体或简化刺激的简单实验范式，但目前尚不清楚这种观点是否准确地概括为包含多个物体和概念的复杂场景的自然主义观看情况。是否有更高层次的（语言和语义意识形式）特征，被整合在对象和场景特征图上以提供意义？为了回答这些问题，我们将结合联合收割机，在参与者观看丰富而复杂的自然场景时，通过脑电图（EEG）和功能性磁共振成像（fMRI）测量大脑活动，以及经过句子嵌入训练的人工神经网络（ANN）和一系列创新的行为任务（使用我的在线行为实验平台https://meadows-research.com收集）。这个DG周期的重点是将视觉表征理论转移到复杂的自然刺激，语义表征和个体差异领域，所有这些都是该领域相对未开发的领域。这将通过三个核心目标来实现：1-在观看自然场景后，精确地跟踪大脑中的何处以及何时建立语义表征。2-揭示与场景语义相关的活动模式如何影响我们的视觉意识内容。3-确定大脑表征的细微差异（即个体之间活动模式可辨别性的变化）和行为（在一系列认知任务中测量）之间的联系。