The role of multisensory redundancy in infant speech perception and language acquisition

婴儿接受的大多数言语输入，无疑也是最吸引人的，来自于面对面的互动，在这种互动中，婴儿既听到又看到正在产生的言语。多感官输入有助于学习和记忆。在（至少是成年人）言语的情况下，大脑似乎有组织地将言语作为双感知（听觉和视觉）而不是单感知（仅听觉）来处理。 指导性假设是言语感知在生命早期是双感知的，因此为了充分理解言语处理如何引导语言习得，不仅需要研究听觉言语感知，还需要研究听觉-视觉（AV）言语感知。两个目标如下。首先是确定婴儿大脑中AV语音处理的神经基础。第二个是探讨婴儿是否使用AV而不仅仅是听觉输入，以最有效地获得越来越复杂的语言方面。 提出了五组4-18个月大的婴儿的实验。在前两种方法中，近红外光谱（NIRS）被用来确定听觉和视觉语音是否独立地激活婴儿大脑中的语言区域。接下来的三组实验，其中涉及眼动跟踪和近红外光谱，测试的假设，即双感官AV表示是最佳的获得越来越复杂的语言水平，通过比较性能在听觉与AV条件。实验组3测试在音节语境中形成音位的等价类的能力。实验组4探讨了婴儿发起的单词映射启发式（链接新的单词到新的对象，没有明确的教学）的结果，在保留该词的上下文。实验集5探讨了韵律（节奏，音高，振幅）信息在母语词序自举习得中的应用。据预测，年龄较小的婴儿将解决这些任务，年龄较大的婴儿将表现出更好的性能，如果他们与AV，而不是听觉语言，专门的大脑语言系统将在AV条件下更加活跃。 如果婴儿的大脑在生命的早期就被设定为以双感知的方式处理语音，如果双感知语音是语言赖以建立的特权表征，那么我们研究语言习得的理论和方法就需要改变。此外，这些结果表明，面对面互动的机会不仅对婴儿的情感发展很重要，而且对最佳感知和学习母语也很重要。这将指出一种特殊的早期干预，用于有感官缺陷的婴儿，反应迟钝的父母，或自闭症等疾病，使他们看起来更少说话的面孔。

Most speech input infants receive, and undoubtedly the most engaging, comes from face-to-face interactions in which the infant both hears and sees speech being produced. Multisensory input facilitates learning and memory. And in the case of (at least adult) speech, the brain appears organized to process speech as bisensory (Auditory and Visual) rather than unisensory (Auditory only).      The guiding hypothesis is that speech perception is bisensory from early in life, and thus to fully understand how speech processing bootstraps language acquisition, it is necessary to study not just Auditory speech perception but also Auditory-Visual (AV) speech perception. Two objectives follow. The first is to determine the neural underpinning of AV speech processing in the infant brain. The second is to explore whether infants use AV rather than just Auditory input to most effectively acquire increasingly complex aspects of language.      Five sets of experiments with 4–18-month-old infants are proposed. In the first two, near infrared spectroscopy (NIRS) is used to determine whether both Auditory and Visual speech independently activate the language areas in the infant brain. The next three sets of experiments, which involve eye tracking and NIRS, test the hypothesis that the bisensory AV representation is optimal for acquiring increasingly complex levels of language by comparing performance in Auditory-only vs. AV conditions. Experimental Set 3 tests the ability to form equivalence classes of phonemes across variation in syllabic context. Experimental Set 4 explores the contexts in which an infant-initiated word-mapping heuristic (linking novel words to novel objects without explicit teaching) results in retention of that word. Experimental Set 5 explores the use of prosodic (rhythm, pitch, amplitude) information in bootstrapping acquisition of the native language’s word order. It is predicted that younger infants will solve these tasks and older infants will show better performance if they are presented with AV rather than Auditory-only speech, and that specialized brain language systems will be more activated in AV conditions.      If the infant brain is set up to process speech as bisensory from early in life, and if bisensory speech is the privileged representation on which language is built, then our theories and methods of studying language acquisition will need to change. Moreover, such results would suggest that the opportunity for face-to-face interaction is important not only for infant emotional development, but also for optimally perceiving and learning the native language. This would point to a particular kind of early intervention for infants with sensory deficits, unresponsive parents, or conditions such as autism that make them look less at talking faces.