The role of multisensory redundancy in infant speech perception and language acquisition

婴儿接受的大多数语言输入，无疑是最吸引人的，来自于面对面的互动，在这种互动中，婴儿既听到又看到正在产生的语言。多感官输入促进学习和记忆。在（至少是成人）语言的情况下，大脑似乎组织起来以双感官（听觉和视觉）而不是单感官（听觉）来处理语言。***指导假设是，语音感知从生命早期开始就是双感觉的，因此，为了充分理解语音处理如何引导语言习得，不仅需要研究听觉语音感知，还需要研究视听语音感知。两个目标随之而来。首先是确定婴儿大脑中AV语音处理的神经基础。二是探索婴儿是否使用视听输入而不仅仅是听觉输入来最有效地获得日益复杂的语言方面。***拟对4-18月龄婴儿进行5组实验。在前两项研究中，近红外光谱（NIRS）被用来确定听觉和视觉语言是否分别激活了婴儿大脑中的语言区域。接下来的三组实验，包括眼动追踪和近红外光谱，通过比较纯听觉和AV条件下的表现，来验证双感官AV表征对于获得越来越复杂的语言水平是最佳的假设。实验集3测试学生在不同的音节语境中形成音素等价类的能力。实验集4探讨了婴儿发起的词映射启发式（在没有明确教学的情况下将新单词与新物体联系起来）导致该单词保留的上下文。实验集5探讨了韵律（节奏、音高、振幅）信息在母语语序习得中的应用。据预测，年龄较小的婴儿可以解决这些任务，而年龄较大的婴儿在使用AV而不是纯听觉语言的情况下会表现得更好，并且专门的大脑语言系统在AV条件下会更活跃。***如果婴儿的大脑在生命早期就被设置为以感知语言来处理语言，如果感知语言是语言构建的特权表征，那么我们研究语言习得的理论和方法就需要改变。此外，这些结果表明，面对面互动的机会不仅对婴儿的情感发展很重要，而且对最佳地感知和学习母语也很重要。这将为有感觉缺陷、父母反应迟钝或自闭症等病症的婴儿提供一种特殊的早期干预，这些病症会使他们少看说话的面孔

Most speech input infants receive, and undoubtedly the most engaging, comes from face-to-face interactions in which the infant both hears and sees speech being produced. Multisensory input facilitates learning and memory. And in the case of (at least adult) speech, the brain appears organized to process speech as bisensory (Auditory and Visual) rather than unisensory (Auditory only).***     The guiding hypothesis is that speech perception is bisensory from early in life, and thus to fully understand how speech processing bootstraps language acquisition, it is necessary to study not just Auditory speech perception but also Auditory-Visual (AV) speech perception. Two objectives follow. The first is to determine the neural underpinning of AV speech processing in the infant brain. The second is to explore whether infants use AV rather than just Auditory input to most effectively acquire increasingly complex aspects of language.***     Five sets of experiments with 4-18-month-old infants are proposed. In the first two, near infrared spectroscopy (NIRS) is used to determine whether both Auditory and Visual speech independently activate the language areas in the infant brain. The next three sets of experiments, which involve eye tracking and NIRS, test the hypothesis that the bisensory AV representation is optimal for acquiring increasingly complex levels of language by comparing performance in Auditory-only vs. AV conditions. Experimental Set 3 tests the ability to form equivalence classes of phonemes across variation in syllabic context. Experimental Set 4 explores the contexts in which an infant-initiated word-mapping heuristic (linking novel words to novel objects without explicit teaching) results in retention of that word. Experimental Set 5 explores the use of prosodic (rhythm, pitch, amplitude) information in bootstrapping acquisition of the native language's word order. It is predicted that younger infants will solve these tasks and older infants will show better performance if they are presented with AV rather than Auditory-only speech, and that specialized brain language systems will be more activated in AV conditions.***     If the infant brain is set up to process speech as bisensory from early in life, and if bisensory speech is the privileged representation on which language is built, then our theories and methods of studying language acquisition will need to change. Moreover, such results would suggest that the opportunity for face-to-face interaction is important not only for infant emotional development, but also for optimally perceiving and learning the native language. This would point to a particular kind of early intervention for infants with sensory deficits, unresponsive parents, or conditions such as autism that make them look less at talking faces.**