The role of multisensory redundancy in infant speech perception and language acquisition

点击翻译按钮获取中文摘要

Most speech input infants receive, and undoubtedly the most engaging, comes from face-to-face interactions in which the infant both hears and sees speech being produced. Multisensory input facilitates learning and memory. And in the case of (at least adult) speech, the brain appears organized to process speech as bisensory (Auditory and Visual) rather than unisensory (Auditory only).      The guiding hypothesis is that speech perception is bisensory from early in life, and thus to fully understand how speech processing bootstraps language acquisition, it is necessary to study not just Auditory speech perception but also Auditory-Visual (AV) speech perception. Two objectives follow. The first is to determine the neural underpinning of AV speech processing in the infant brain. The second is to explore whether infants use AV rather than just Auditory input to most effectively acquire increasingly complex aspects of language.      Five sets of experiments with 4–18-month-old infants are proposed. In the first two, near infrared spectroscopy (NIRS) is used to determine whether both Auditory and Visual speech independently activate the language areas in the infant brain. The next three sets of experiments, which involve eye tracking and NIRS, test the hypothesis that the bisensory AV representation is optimal for acquiring increasingly complex levels of language by comparing performance in Auditory-only vs. AV conditions. Experimental Set 3 tests the ability to form equivalence classes of phonemes across variation in syllabic context. Experimental Set 4 explores the contexts in which an infant-initiated word-mapping heuristic (linking novel words to novel objects without explicit teaching) results in retention of that word. Experimental Set 5 explores the use of prosodic (rhythm, pitch, amplitude) information in bootstrapping acquisition of the native language’s word order. It is predicted that younger infants will solve these tasks and older infants will show better performance if they are presented with AV rather than Auditory-only speech, and that specialized brain language systems will be more activated in AV conditions.      If the infant brain is set up to process speech as bisensory from early in life, and if bisensory speech is the privileged representation on which language is built, then our theories and methods of studying language acquisition will need to change. Moreover, such results would suggest that the opportunity for face-to-face interaction is important not only for infant emotional development, but also for optimally perceiving and learning the native language. This would point to a particular kind of early intervention for infants with sensory deficits, unresponsive parents, or conditions such as autism that make them look less at talking faces.