Deep learning-based image deconstruction method with maintained saliency
Deep learning-based image deconstruction method with maintained saliency
复制标题
DOI：
10.1016/j.neunet.2022.08.015
复制
发表时间：
2022-08
期刊：
影响因子：
--
通讯作者：
Keisuke Fujimoto;Kojiro Hayashi;Risa Katayama;Sehyung Lee;Zhen Liang-;W. Yoshida;Shin Ishii
中科院分区：
文献类型：
--
作者：
Keisuke Fujimoto;Kojiro Hayashi;Risa Katayama;Sehyung Lee;Zhen Liang-;W. Yoshida;Shin Ishii
主要吸引自下而上注意力的视觉属性统称为显着性。在这项研究中，为了了解自上而下和自下而上视觉注意中涉及的神经活动，我们的目标是准备具有共同显着性的自然和非自然图像对。为此，我们提出了一种基于深度神经网络的图像变换方法，该方法可以生成新图像，同时保持一致的特征图，特别是显着图。这是一个不适定问题，因为从图像到其对应的特征图的变换可能是多对一的，并且在我们的特定情况下，各种图像将共享相同的显着图。虽然随机图像生成具有解决这种不适定问题的潜力，但大多数现有方法都集中在增加整体风格/触摸信息的多样性，同时保持所生成图像的自然性。为此，我们开发了一种新的图像变换方法，该方法采用了更高维的潜变量，使生成的图像看起来不自然，上下文信息较少，但保留了高度多样性的局部图像结构。虽然这样的高维潜在空间很容易崩溃，我们提出了一个新的正则化的基础上Kullback-Leibler分歧，以避免崩溃的潜在分布。我们还使用我们新准备的自然和相应的非自然图像进行了人体实验，以测量明显的眼球运动和功能性磁共振成像，并发现这些图像诱导了与自上而下和自下而上注意力处理相关的独特神经活动。
Visual properties that primarily attract bottom-up attention are collectively referred to as saliency. In this study, to understand the neural activity involved in top-down and bottom-up visual attention, we aim to prepare pairs of natural and unnatural images with common saliency. For this purpose, we propose an image transformation method based on deep neural networks that can generate new images while maintaining the consistent feature map, in particular the saliency map. This is an ill-posed problem because the transformation from an image to its corresponding feature map could be many-to-one, and in our particular case, the various images would share the same saliency map. Although stochastic image generation has the potential to solve such ill-posed problems, the most existing methods focus on adding diversity of the overall style/touch information while maintaining the naturalness of the generated images. To this end, we developed a new image transformation method that incorporates higher-dimensional latent variables so that the generated images appear unnatural with less context information but retain a high diversity of local image structures. Although such high-dimensional latent spaces are prone to collapse, we proposed a new regularization based on Kullback–Leibler divergence to avoid collapsing the latent distribution. We also conducted human experiments using our newly prepared natural and corresponding unnatural images to measure overt eye movements and functional magnetic resonance imaging, and found that those images induced distinctive neural activities related to top-down and bottom-up attentional processing.