Decoding visual content and perception from neuronal population activity in visual cortex: VSDI, fMRI and computational modeling

视觉刺激是如何由神经元群体反应编码的，以及它们如何被视觉系统的后续阶段解码，这是迄今为止仍然悬而未决的基本问题。我们的联盟提出了一种非常新颖的跨学科方法，将我们在多尺度脑活动测量方面的独特专业知识和协同科学技能与我们在大脑建模、神经计算和计算机视觉方面的专业知识相结合。我们建议研究神经元群体反应是如何编码视觉信息的，以及这些信息是如何被随后的大脑区域“读出”的，以便调节视觉感知和行为。首先，我们计划在麻醉的动物的视皮层中，使用多种分辨率的大脑活动测量：电压敏感染料(VSD)和功能磁共振成像信号，对群体反应中的刺激进行解码(读出)，表现为猴子和人类。VSD成像以高空间和时间分辨率同时测量神经群的反应，而功能磁共振成像提供了更广泛的大脑复盖面，但空间和时间分辨率降低。基于这些信号，我们的目标是开发和比较用于详细重建从基本刺激特征到自然图像和电影的视觉刺激的解码规则。比较将提供关于不同测量方式和物种的刺激编码的信息。此外，我们将把最先进的计算机视觉技术与脑信号的多尺度建模相结合，以促进解码性能。接下来，我们将研究神经表示和解码规则是如何在视觉通路上改变的，以及大脑是否使用不同的解码规则来实现更高的视觉功能。实现我们的目标将使我们能够更好地理解和模拟大脑信号，并可能促进医学的发展

How visual stimuli are encoded by neuronal population responses and how they can be decoded by subsequent stages of the visual system are fundamental questions that are to date still open. Our consortium proposes a highly novel, interdisciplinary approach, in which we combine our unique expertise and synergistic scientific skills in multi-scale brain activity measurements with our expertise in brain modeling, neural computation and computer vision. We suggest studying how neuronal population responses encode visual information and how this information is "read out" by subsequent brain areas in order to mediate visual perception and behavior. First, we plan to decode ("read out") stimuli from population responses in the visual cortex of anaesthetized animals, behaving monkeys and humans using brain activity measurements at multiple resolutions: voltage-sensitive dye (VSD) and fMRI signals. VSD imaging measures neural population responses at high spatial and temporal resolution simultaneously whereas fMRI provides extended coverage of the brain but with decreased spatial and temporal resolution. Based on these signals we aim to develop and compare decoding rules that are used to reconstruct in detail visual stimuli ranging from basic stimulus features to natural images and movies. The comparison will provide information on stimulus encoding across measurement modalities and species. In addition, we will combine state-of-the-art computer vision techniques with multi-scale modeling of brain signals to facilitate decoding performance. Next, we will study how neural representation and decoding rules change along the visual pathway and whether the brain uses different decoding rules for higher visual functions. Achieving our goals will allow better understanding and modeling of brain signals and may facilitate the development of medical