Neural substrates of optimal multisensory integration

点击翻译按钮获取中文摘要

Project Summary When communicating face-to-face, humans receive information in two sensory modalities: visual information from the face of the talker; and auditory information from the voice of the talker. The pandemic has brought into sharp focus the importance of audiovisual speech: mask wearing obscures the talker's face while muffling the voice, a double whammy that hinders communication. The popularity of video conferencing software attests to the importance placed by people on seeing a talker's face as well as hearing their voice. Although audiovisual speech perception is very important, we know little about the neural mechanisms for this uniquely human ability. We will remedy this gap in knowledge using most powerful techniques in human neuroscience: computational modeling; behavioral studies; ultra-high-field (7 tesla) functional MRI (fMRI); and the examination of epilepsy patients who have electrodes implanted in their brain for the treatment of medically intractable epilepsy, a technique referred to as intracranial electroencephalography (iEEG). The anatomical focus of the proposal is the posterior superior temporal sulcus/gyrus (STS/G), known since the time of Wernicke to be important for speech perception. The behavioral focus of our proposal is the new discovery that a classic audiovisual speech illusion, known as the McGurk effect, can produce dramatic, long-lasting changes in auditory-only speech perception, turning a ba into a da. This phenomenon, termed fusion-induced recalibration (FIR), provides a tool to advance computational and neural studies of speech perception. The first aim will develop computational models of speech perception and test against behavioral data. Different models will be fit to speech perception data before, during and after exposure to the McGurk effect. Fitted models will be compared using held-out behavioral data. Because the models instantiate different theoretical constructs, model comparison will determine which explanatory constructs are essential. These results will provide a solid theoretical grounding for future studies, including those in Aims 2 and 3: searching for a neural correlate of an unjustified construct is likely to be fruitless. The second aim will examine speech perceptin through the lens of patterns of activity in STS/G measured with 7 tesla fMRI. We expect to observe reliable changes in STS/G response patterns before and after exposure to the McGurk effect, reflecting modification of speech representations (in contrast, in cortical areas driven solely by acoustic features, McGurk exposure should not change fMRI response patterns.) The third aim will use iEEG to record broadband high-frequency activity (BHA) from small populations of STS/G neurons with high temporal resolution. Responses to the auditory- only component of the McGurk speech (but not control speech) are predicted to show sustained decreases after successive blocks of audiovisual McGurk exposure, in lockstep with the perceptual development of FIR.