Temporal Information Guided Generative Adversarial Networks for Stimuli Image Reconstruction from Human Brain Activities
Temporal Information Guided Generative Adversarial Networks for Stimuli Image Reconstruction from Human Brain Activities
复制标题
用于人脑活动刺激图像重建的时间信息引导生成对抗网络
DOI：
10.1109/tcds.2021.3098743
复制
发表时间：
2021
影响因子：
5
通讯作者：
Daoqiang Zhang
中科院分区：
文献类型：
--
作者：
Shuo Huang;Liang Sun;Muhammad Yousefnezhad;Meiling Wang;Daoqiang Zhang
了解人类大脑如何工作已经在神经科学和机器学习领域引起了越来越多的关注。以前的研究使用自动编码器和生成对抗网络（GAN）来提高功能磁共振成像（fMRI）数据的刺激图像重建质量。然而，这些方法主要集中于获取两种不同模态的数据之间的相关特征，即，刺激图像和fMRI，而忽略了fMRI数据的时间信息，从而导致次优性能。为了解决这个问题，在这篇文章中，我们提出了一个时间信息引导的GAN（TIGAN）重建视觉刺激从人脑活动。具体而言，该方法由三个关键组件组成，包括：1）用于将刺激图像映射到潜在空间的图像编码器; 2）用于fMRI特征映射的长短期记忆（LSTM）生成器，用于捕获fMRI数据中的时间信息;以及3）用于图像重建的编码器，用于使重建图像与原始图像更相似。此外，为了更好地测量两种不同模态的数据（即，fMRI和自然图像），我们利用成对排序损失来对刺激图像和fMRI进行排序，以确保强相关的对位于顶部，弱相关的对位于底部。在真实数据集上的实验结果表明，与几种最先进的图像重建方法相比，所提出的TIGAN具有更好的性能。
Understanding how the human brain works has attracted increasing attention in both fields of neuroscience and machine learning. Previous studies use autoencoder and generative adversarial networks (GANs) to improve the quality of stimuli image reconstruction from functional magnetic resonance imaging (fMRI) data. However, these methods mainly focus on acquiring relevant features between two different modalities of data, i.e., stimuli images and fMRI, while ignoring the temporal information of fMRI data, thus leading to suboptimal performance. To address this issue, in this article, we propose a temporal information-guided GAN (TIGAN) to reconstruct visual stimuli from human brain activities. Specifically, the proposed method consists of three key components, including: 1) an image encoder for mapping the stimuli images into latent space; 2) a long short-term memory (LSTM) generator for fMRI feature mapping, which is used to capture temporal information in fMRI data; and 3) a discriminator for image reconstruction, which is used to make the reconstructed image more similar to the original image. In addition, to better measure the relationship of two different modalities of data (i.e., fMRI and natural images), we leverage a pairwise ranking loss to rank the stimuli images and fMRI to ensure strongly associated pairs at the top and weakly related ones at the bottom. The experimental results on real-world data sets suggest that the proposed TIGAN achieves better performance in comparison with several state-of-the-art image reconstruction approaches.