Recording of Brain Activity for Speech Processing Usin Audio-Visual Stimuli.

本研究旨在建立一个记录大脑活动(ERP或MEG)的系统，用于词义的语义加工。选择了基本的300个具体名词，并为这些词准备了数字化语音数据、字符(汉字和平假名)的图像数据和图片数据。构建了一个具有再生时序控制的视听刺激同步呈现系统。还准备了一组动态视频刺激，对音频和视频信号进行反向定时操作，以考察视听整合的时间过程。通过使用这些刺激，进行了以下初步实验：比较词义的视觉和听觉加工，并分析视听整合的过程。1.文字和图片刺激的范畴判断的比较。动物名称词作为标准刺激呈现，植物名称词作为稀有刺激呈现。任务是清点植物的名字。图像刺激的事件相关电位在大约400毫秒的VST处表现出特征的正信号。2.字符和位置刺激的视听失配的比较。听觉和视觉刺激同时呈现，不匹配的样本很少呈现。文字刺激与图片刺激的P300潜伏期差异有统计学意义。3.视听言语整合分析。通过测量电磁神经活动来检测视听语音刺激时序的罕见变化，来研究与视听线索绑定有关的神经过程。这些结果表明，存在一种机制，通过绑定听觉和视觉皮层中平行但时空分布的过程来确保连贯的语音分类。

The present study aims at establishing a system for recording brain activities (ERP or MEG) for the semantic processing of word meaning. The basic 300 concrete nouns were selected and for these words digitized speech data image data of the characters (Kanji and Hirakana) and the picture data were prepared. A system was constructed for simultareons presentation of audio-visual stimuli with regnired timing control. A set of dynamic video stimuli with antificial timing manipulation of the audio and video signals were also prepared to examine the temporal process of the audio-visual integration. By using these stimuli, following preliminary experiments were conducted to compare the visual and auditory processing of the word meaning and also to analyze the process of audio-visual integration.1.Comparison of categorical judgment for character and picture stimuli. The words for animal name were presented as standard stimmuli and the words for plant name were presented as rare stimuli. The task was to count the plant name. ERP for the picture stimuli shows characteristic positive shitt at about 400 msec VST.2.Cpmparison of Audio-Visual mismatch for character and pocture stimuli. Audio and visual stimuli were presented simultaneonsly and mismatched samples were presented infrequently. There was a significant difference in the P 300 latency between charactera nd picture stimuli.3.Analyzes of audio-visual speech integration. Neural processes relating to audio-visual cue binding were investigated by measuring the electro-magnetic neural activities detecting infrequent changes in the timing of audio-visual speech stimuli. The results suggest the existence of a mechanism which guarntees coherent phonetic categorization by binding parallel but spatio-temporally distributing processes in the auditory and visual cortices.