Attentionally modulated subjective images reconstructed from brain activity
Attentionally modulated subjective images reconstructed from brain activity
复制标题
DOI：
10.1101/2020.12.27.424510
复制
发表时间：
2020-12
期刊：
影响因子：
--
通讯作者：
T. Horikawa;Y. Kamitani
中科院分区：
文献类型：
--
作者：
T. Horikawa;Y. Kamitani
从大脑活动中重建的视觉图像产生的图像，其特征与给定任意视觉实例的视觉皮层中的神经表征一致[1-3]，可能反映了人的视觉体验。以往的重构研究关注的是刺激图像如何被忠实地重构，或者在没有外部刺激的情况下，心理想象的内容是否可以被重构。然而，许多视觉研究表明，即使是刺激感知也是由刺激诱导的过程和自上而下的过程形成的。特别是，注意力（或缺乏注意力）会深刻影响视觉体验[4-8]和大脑活动[9-21]。在这里，为了研究自上而下的注意力如何影响视觉图像的神经表征和重建，我们使用最先进的方法（深度图像重建[3]）从fMRI活动中重建视觉图像，当受试者注意到两个图像中的一个叠加有相等的加权对比度时测量。深度图像重建利用大脑和深度神经网络（DNN）之间的分层对应关系，将大脑活动转换（解码）为多层的DNN特征，然后创建与解码的DNN特征一致的图像[3，22，23]。使用在对单个自然图像的fMRI响应上训练的深度图像重建模型，我们解码了注意力试验期间的大脑活动。行为评估表明，重建类似于出席，而不是无人值守的图像。重建可以通过叠加图像建模，其对比度偏向于参加者，这与在单独的会话中测量的注意下的刺激的外观相当。注意力调制存在于广泛的层次视觉表征中，并反映了大脑-DNN的对应关系。我们的研究结果表明，自上而下的注意计数器刺激诱导的反应，并调制神经表征呈现重建根据主观外观。这些重建似乎反映了视觉体验和意志控制的内容，为基于大脑的交流和创造开辟了新的可能性。
Visual image reconstruction from brain activity produces images whose features are consistent with the neural representations in the visual cortex given arbitrary visual instances [1–3], presumably reflecting the person’s visual experience. Previous reconstruction studies have been concerned either with how stimulus images are faithfully reconstructed or with whether mentally imagined contents can be reconstructed in the absence of external stimuli. However, many lines of vision research have demonstrated that even stimulus perception is shaped both by stimulus-induced processes and top-down processes. In particular, attention (or the lack of it) is known to profoundly affect visual experience [4–8] and brain activity [9–21]. Here, to investigate how top-down attention impacts the neural representation of visual images and the reconstructions, we use a state-of-the-art method (deep image reconstruction [3]) to reconstruct visual images from fMRI activity measured while subjects attend to one of two images superimposed with equally weighted contrasts. Deep image reconstruction exploits the hierarchical correspondence between the brain and a deep neural network (DNN) to translate (decode) brain activity into DNN features of multiple layers, and then create images that are consistent with the decoded DNN features [3, 22, 23]. Using the deep image reconstruction model trained on fMRI responses to single natural images, we decode brain activity during the attention trials. Behavioral evaluations show that the reconstructions resemble the attended rather than the unattended images. The reconstructions can be modeled by superimposed images with contrasts biased to the attended one, which are comparable to the appearance of the stimuli under attention measured in a separate session. Attentional modulations are found in a broad range of hierarchical visual representations and mirror the brain–DNN correspondence. Our results demonstrate that top-down attention counters stimulus-induced responses and modulate neural representations to render reconstructions in accordance with subjective appearance. The reconstructions appear to reflect the content of visual experience and volitional control, opening a new possibility of brain-based communication and creation.