Dynamic Interaction of Multisensory Information in 3D Object Recognition

本研究的目的是通过心理物理和功能磁共振成像脑活动成像实验，探讨视觉和体感信息对三维物体识别的动态脑机制。在本研究中，我们利用虚拟和真实的三维物体开发了新颖的实验方法，并通过心理物理实验揭示了视觉和体感相互作用的合作和竞争机制，并测量了大脑活动，以发现参与视觉和触觉信息的三维物体学习和图像生成的大脑区域。针对视觉和体感相互作用的合作和竞争机制，我们利用视觉-触觉显示系统对模糊视觉模式的解释进行了心理物理实验，发现触觉信息影响人的视觉感知，双眼视差信息在三维视觉和触觉形状估计中发挥重要作用。我们进一步开发了多感官互动系统，通过设计新的算法来实时集成3D图像，力反馈和接触声音。在人类物体识别的fMRI实验中，我们使用3D打印机生成的真实物体，并测量受试者在执行触觉物体学习和自上而下图像生成任务时的大脑活动。结果表明，顶叶下小叶参与触觉三维物体学习，BA7和前叶区参与触觉自上而下判断。我们进一步进行了跨模态自顶向下加工实验，结果表明枕侧复合体（LOC）表征的视觉物体形状可以从顶叶下小叶（BA40）获得的触觉物体记忆中提取，体感区表征的触觉物体形状可以从梭状回获得的视觉物体记忆中提取。结果还表明，BA7及周边区域可呈现多感官影像。基于这些发现，我们建立了一个人类多感觉整合的大脑网络模型。少

The purpose of this study is to investigate dynamic brain mechanisms of 3D object recognition with visual and somatosensory information through psychophysical and fMRI brain activity imaging experiments. In this study, we developed novel experimental methods using virtual and real 3D objects, and conducted psychophysical experiments to reveal cooperative and competitive mechanisms of visual and somatosensory interaction, and measured brain activities to find brain regions involved in 3D object learning and image generation with visual and haptic information.Regarding cooperative and competitive mechanisms of visual and somatosensory interaction, we conducted psychophysical experiments on the interpretation of ambiguous visual patterns using a visual-haptic display system and found that haptic information affects human visual perception and that binocular disparity information plays an important role in 3D visual and haptic shape estimation. We further developed multi-sensory interactio … More n system by devising new algorithms to integrate 3D image, force feedback, and contact sounds in real-time.In fMRI experiments on human object recognition, we used real objects produced by a 3D printer and measured brain activities when the subjects performed haptic object learning and top-down image generation tasks. The results suggested that the inferior parietal lobule is involved in haptic 3D object learning and BA7 and the precuneous regions are involved in haptic top-down judgment. We further conducted experiments on cross-modal top-down processing, showing that the visual object shape represented in the lateral occipital complex(LOC) could be retrieved from haptic object memory acquired in the inferior parietal lobule(BA40), and that the haptic object shape represented in the somatosensory areas could be retrieved from visual object memory acquired in the fusiform gyrus. The results also suggested that multi-sensory imagery could be represented in BA7 and the precuneous regions. Based on these findings, we developed a brain network model of human multi-sensory integration. Less