Use of lip-read information in speech perception and its relation to second language proficiency

McGurk效应是一种视听错觉，它表明在语音感知过程中，视觉唇形信息与相互冲突的听觉信息相融合，表明语音感知在面对面交流中不仅仅是一个听觉过程，而是一个多模态过程。我们之前的研究表明，麦格克效应在以日语和美式英语为母语的人的外语听觉演讲中更强，尽管视觉效果在日本人身上比在美国人身上弱。本研究对这些影响视听语音感知的语言因素进行了研究。在实验1中，以汉语为母语的人接受日语和英语刺激。麦格克效应和我们之前测试的日本受试者一样弱，这表明中国受试者的加工方式也有听音偏见。这些在中国读完大学后住在日本的研究对象显示，他们在日本度过的时间与麦格克效应的大小呈正相关。在实验2中，我们通过对母语为汉语和日语的人进行中文和日语刺激测试，验证了McGurk效应在外语听觉言语中更强的“外语效应假说”。结果表明，该假设仅对日本受试者成立，而中国受试者在日语和汉语刺激之间没有表现出任何差异。实验3通过对46名母语为汉语的人进行测试，考察第二语言（日语）熟练程度对麦格克效应的影响。研究结果为这种影响提供了一些证据，但在得出明确结论之前，还需要控制年龄等额外变量。

The McGurk effect is an audiovisual illusion which shows that visual lip information is integrated with conflicting auditory information during speech perception, demonstrating that speech perception is not a solely auditory process in face-to-face communication, but a multimodal process. Our previous research has shown that the McGurk effect is stronger for auditory speech in foreign languages both in native speakers of Japanese and American English, although the visual effect is weaker for the Japanese than for the Americans. The present study investigated such linguistic factors in audiovisual speech perception.In Experiment 1, native speakers of Chinese were tested with Japanese and English stimuli. The McGurk effect was as weak as for our previously tested Japanese subjects, suggesting an audition-biased manner of processing in the Chinese subjects as well. These subjects, who lived in Japan after finishing college in China, showed a positive correlation between the time they had spent in Japan and the size of the McGurk effect.In Experiment 2, we tested the "foreign language effect hypothesis" that the McGurk effect is stronger for auditory speech in foreign languages, by testing native speakers of Chinese and Japanese with Chinese and Japanese stimuli. The results showed that the hypothesis was true only for the Japanese subjects and that the Chinese subjects did not show any differences between the Japanese and Chinese stimuli.Experiment 3 examined the effect of second language (Japanese) proficiency on the McGurk effect, by testing forty six native speakers of Chinese. The results suggested some evidence for such an effect, but there were extra variables such as age which should be controlled before a definite conclusion.