Integration of auditory and visual information in human communication

我们使用异步性检测任务考察了人类口头交流中听觉和视觉信息之间的相互作用。我们发现了异步性检测中的一些文化因素，如说话人和听话人的母语、面部特征等。句子意义的可理解性似乎对异步性检测也很重要。我们还发现了当视听信息不同步时，声音感知音量减小的现象。我们对视觉信息对语音理解的贡献的实验表明，视觉信息对语音认知的利用存在较大的个体差异。

We examined the interactions between audio and visual information in human oral communications using the asynchrony detection tasks. We found some cultural factors in asynchrony detection such as native language of speakers and listeners, facial features, and so on. The understandability of the meaning of sentences are also seemed to be important for asynchrony detection. We also found the phenomenon in which the percepted volume of sound decrease when the audio and visual information is no synchronized. Our experiments on the contributions of the visual information to speech understandings shows there is large individual differences in utilization of visual information to speech cognition.