Psychological experiment for clarifying the neural mechanism of complementary interaction between visual and haptic information in their binding process

当一个人用手观察一个物体时，视觉往往主导着视觉-触觉的综合感知，但在某些情况下，这一过程明显受到触觉的影响。本研究的目的是在心理实验的基础上，探讨视觉和触觉信息在结合过程中互补相互作用的神经机制。心理实验我们进行了心理实验，用触觉设备和3d视觉系统构建虚拟空间。心理实验的任务如下：被试通过操作触觉装置感受呈现在电脑屏幕上的立方体的重量，然后以同样的方式感受另一个立方体的重量，这个立方体的视觉特征（如大小、颜色）与第一次呈现的立方体不同。受试者必须决定哪个立方体的权重更大。我们考察了视觉字符对体重辨别任务的影响。实验结果表明，立方体的视觉特征会引起识别任务的显著变化，这支持了视觉信息被快速加工并在视觉和触觉信息的结合过程中充当预测信号的假设。为了探讨感官信息加工的神经机制，我们建立了视觉、听觉和电感觉的神经模型。在建模研究的基础上，建立了视觉与触觉信息结合过程的计算模型。基于反向传播网络建立了视觉系统的网络模型，网络的输出作为预测信号影响触觉系统的网络。计算模型定性地再现了我们在第1节中描述的心理数据。

When a person looks at an object while exploring it with their hand, vision frequently dominates the integrated visual-haptic perception, but in some circumstances the process is clearly affected by haptics. The aim of this study is to investigate the neural mechanism of the complementary interaction between visual and haptic information in their binding process, based on psychological experiment.1.Psychological experimentsWe conducted psychological experiments in which virtual spaces were constructed with a haptic device and 3D-visual system. The task of the psychological experiment is as follow : The subject feels the weight of a cube presented on a computer screen by manipulating the haptic device, and then feels the weight of an another cube in the same way, which has different visual characters (i.e.size, color) from the cube firstly presented. The subject has to make a decision of which cube has larger weight. We examined the effect of the visual characters on the weight-discrimination task. We showed that the visual characters of cube cause significant changes in the discrimination task, supporting a hypothesis that the visual information is rapidly processed and then act as a prediction signal in binding process of visual and haptic information.2.Neural network model for binding visual and haptic informationTo investigate the neural mechanism of sensory information processing, we made neural models for visual, auditory, and electrosensory systems. Based on the modeling study, we made a computational model for binding process of visual and haptic information. The network model of visual system was made based on backpropagation network, and the output of the network affects the network of haptic system as a prediction signal. The computational model reproduced qualitatively our psychological data described in section 1.