Use of lip-read information in speech perception and its relation to second language proficiency

McGurk效应是一种视听错觉，它表明在言语感知过程中，视觉嘴唇信息与相互冲突的听觉信息被整合在一起，这表明言语感知不是面对面交流中的一个单纯的听觉过程，而是一个多通道的过程。我们之前的研究表明，在以日语和美国英语为母语的人中，外语听觉言语的McGurk效应更强，尽管日语人的视觉效果比美国人弱。本研究探讨了视听言语知觉中的语言因素。实验一对以汉语为母语的人进行了日语和英语刺激的测试。麦格克效应和我们之前测试的日语受试者一样弱，这表明中国受试者的加工方式也是试听偏向的。这些被试在中国大学毕业后居住在日本，他们在日本的时间与麦格克效应的大小呈正相关。在实验2中，我们通过测试母语为汉语和日语的人在汉语和日语刺激下的麦格克效应，验证了外语中麦格克效应较强的“外语效应假说”。实验三通过对46名母语为汉语的受试者进行测试，考察了第二语言水平对McGurk效应的影响。研究结果为这种效应提供了一些证据，但在得出明确结论之前，还有一些额外的变量，如年龄，应该加以控制。

The McGurk effect is an audiovisual illusion which shows that visual lip information is integrated with conflicting auditory information during speech perception, demonstrating that speech perception is not a solely auditory process in face-to-face communication, but a multimodal process. Our previous research has shown that the McGurk effect is stronger for auditory speech in foreign languages both in native speakers of Japanese and American English, although the visual effect is weaker for the Japanese than for the Americans. The present study investigated such linguistic factors in audiovisual speech perception.In Experiment 1, native speakers of Chinese were tested with Japanese and English stimuli. The McGurk effect was as weak as for our previously tested Japanese subjects, suggesting an audition-biased manner of processing in the Chinese subjects as well. These subjects, who lived in Japan after finishing college in China, showed a positive correlation between the time they had spent in Japan and the size of the McGurk effect.In Experiment 2, we tested the "foreign language effect hypothesis" that the McGurk effect is stronger for auditory speech in foreign languages, by testing native speakers of Chinese and Japanese with Chinese and Japanese stimuli. The results showed that the hypothesis was true only for the Japanese subjects and that the Chinese subjects did not show any differences between the Japanese and Chinese stimuli.Experiment 3 examined the effect of second language (Japanese) proficiency on the McGurk effect, by testing forty six native speakers of Chinese. The results suggested some evidence for such an effect, but there were extra variables such as age which should be controlled before a definite conclusion.