The role of multisensory redundancy in infant speech perception and language acquisition

婴儿接受的大多数语言输入，无疑是最吸引人的，来自面对面的互动，在这种互动中，婴儿既能听到也能看到语言的产生。多感官输入有助于学习和记忆。在言语(至少是成人)的情况下，大脑似乎被组织成双感觉(听觉和视觉)而不是单感觉(仅听觉)来处理语言。*说明性假说是言语知觉从小就是双感觉的，因此为了充分理解言语处理如何引导语言习得，不仅有必要研究听觉言语知觉，而且有必要研究听觉-视觉(AV)言语知觉。两个目标随之而来。第一个是确定婴儿大脑中AV语音处理的神经基础。第二个是探索婴儿是否使用听觉输入而不仅仅是听觉输入来最有效地获得日益复杂的语言方面。*建议对4-18个月大的婴儿进行五组实验。在前两种方法中，近红外光谱(NIRS)被用来确定听觉和视觉语言是否都独立地激活了婴儿大脑中的语言区域。接下来的三组实验涉及眼球跟踪和近红外光谱，通过比较纯听觉和听觉条件下的表现来检验这一假设，即双感觉AV表征对于获得日益复杂的语言水平是最佳的。实验3测试了在音节上下文中形成不同音素对等类的能力。实验4探索了婴儿启动的单词映射启发式(在没有明确教学的情况下将新单词与新对象联系在一起)导致该单词保持的情景。实验5探讨了韵律(节奏、音高、幅度)信息在母语词序自举习得中的应用。据预测，较小的婴儿将解决这些任务，较大的婴儿将表现出更好的表现，如果他们被呈现给AV而不是仅限于听觉的语音，而且专门的大脑语言系统在AV条件下将更加激活。*如果婴儿的大脑在生命早期就被设置为以双感觉来处理语音，如果双感觉语言是语言构建的特权表征，那么我们研究语言习得的理论和方法将需要改变。此外，这些结果表明，面对面互动的机会不仅对婴儿的情感发展很重要，而且对以最佳方式感知和学习母语也很重要。这将为患有感觉缺陷、父母反应迟钝或自闭症等疾病的婴儿提供一种特殊的早期干预，使他们更少地看着会说话的脸。

Most speech input infants receive, and undoubtedly the most engaging, comes from face-to-face interactions in which the infant both hears and sees speech being produced. Multisensory input facilitates learning and memory. And in the case of (at least adult) speech, the brain appears organized to process speech as bisensory (Auditory and Visual) rather than unisensory (Auditory only).***     The guiding hypothesis is that speech perception is bisensory from early in life, and thus to fully understand how speech processing bootstraps language acquisition, it is necessary to study not just Auditory speech perception but also Auditory-Visual (AV) speech perception. Two objectives follow. The first is to determine the neural underpinning of AV speech processing in the infant brain. The second is to explore whether infants use AV rather than just Auditory input to most effectively acquire increasingly complex aspects of language.***     Five sets of experiments with 4-18-month-old infants are proposed. In the first two, near infrared spectroscopy (NIRS) is used to determine whether both Auditory and Visual speech independently activate the language areas in the infant brain. The next three sets of experiments, which involve eye tracking and NIRS, test the hypothesis that the bisensory AV representation is optimal for acquiring increasingly complex levels of language by comparing performance in Auditory-only vs. AV conditions. Experimental Set 3 tests the ability to form equivalence classes of phonemes across variation in syllabic context. Experimental Set 4 explores the contexts in which an infant-initiated word-mapping heuristic (linking novel words to novel objects without explicit teaching) results in retention of that word. Experimental Set 5 explores the use of prosodic (rhythm, pitch, amplitude) information in bootstrapping acquisition of the native language's word order. It is predicted that younger infants will solve these tasks and older infants will show better performance if they are presented with AV rather than Auditory-only speech, and that specialized brain language systems will be more activated in AV conditions.***     If the infant brain is set up to process speech as bisensory from early in life, and if bisensory speech is the privileged representation on which language is built, then our theories and methods of studying language acquisition will need to change. Moreover, such results would suggest that the opportunity for face-to-face interaction is important not only for infant emotional development, but also for optimally perceiving and learning the native language. This would point to a particular kind of early intervention for infants with sensory deficits, unresponsive parents, or conditions such as autism that make them look less at talking faces.**