Decoding visual content and perception from neuronal population activity in visual cortex: VSDI, fMRI and computational modeling

视觉刺激是如何被神经元群体反应编码的，以及它们如何被视觉系统的后续阶段解码，这是迄今为止仍未解决的基本问题。我们的联盟提出了一种高度新颖的跨学科方法，将我们在多尺度大脑活动测量方面的独特专业知识和协同科学技能与我们在大脑建模，神经计算和计算机视觉方面的专业知识结合起来。我们建议研究神经元群体的反应如何编码视觉信息，以及这些信息如何被随后的大脑区域“读出”，以调节视觉感知和行为。首先，我们计划解码（“读出”）来自麻醉动物、行为正常的猴子和人类视觉皮层的群体反应的刺激，使用多种分辨率的大脑活动测量：电压敏感染料（VSD）和功能磁共振成像信号。VSD成像同时以高空间和时间分辨率测量神经群体反应，而fMRI提供了对大脑的扩展覆盖，但空间和时间分辨率降低。基于这些信号，我们的目标是开发和比较解码规则，用于重建详细的视觉刺激，从基本的刺激特征到自然图像和电影。比较将提供跨测量方式和物种的刺激编码信息。此外，我们将结合最先进的计算机视觉技术与大脑信号的多尺度建模，以促进解码性能。接下来，我们将研究神经表征和解码规则如何沿着视觉通路变化，以及大脑是否使用不同的解码规则来实现更高的视觉功能。实现我们的目标将有助于更好地理解和模拟大脑信号，并可能促进医学的发展

How visual stimuli are encoded by neuronal population responses and how they can be decoded by subsequent stages of the visual system are fundamental questions that are to date still open. Our consortium proposes a highly novel, interdisciplinary approach, in which we combine our unique expertise and synergistic scientific skills in multi-scale brain activity measurements with our expertise in brain modeling, neural computation and computer vision. We suggest studying how neuronal population responses encode visual information and how this information is "read out" by subsequent brain areas in order to mediate visual perception and behavior. First, we plan to decode ("read out") stimuli from population responses in the visual cortex of anaesthetized animals, behaving monkeys and humans using brain activity measurements at multiple resolutions: voltage-sensitive dye (VSD) and fMRI signals. VSD imaging measures neural population responses at high spatial and temporal resolution simultaneously whereas fMRI provides extended coverage of the brain but with decreased spatial and temporal resolution. Based on these signals we aim to develop and compare decoding rules that are used to reconstruct in detail visual stimuli ranging from basic stimulus features to natural images and movies. The comparison will provide information on stimulus encoding across measurement modalities and species. In addition, we will combine state-of-the-art computer vision techniques with multi-scale modeling of brain signals to facilitate decoding performance. Next, we will study how neural representation and decoding rules change along the visual pathway and whether the brain uses different decoding rules for higher visual functions. Achieving our goals will allow better understanding and modeling of brain signals and may facilitate the development of medical