The role of amplitude envelope in auditory-visual integration

点击翻译按钮获取中文摘要

Ventriloquists routinely exploit the perceptual system's propensity to integrate auditory and visual information, using it to trick audiences into "hearing" puppets speak. I have shown that some musicians (percussionists) use similar principles, allowing their audiences to hear certain musical notes as longer or shorter than their acoustic duration. This novel illusion using naturalistic percussion sounds is at odds with earlier research (often using unnatural, artificial sounds), which has repeatedly concluded that such patterns of visual influence are not possible. Curiously, although artificial sounds are popular in a wide range of auditory research, they rarely occur in the natural world. In contrast, percussive sounds are common in our everyday experience. Therefore this proposed research program will explore the degree to which previous work using artificial sounds generalizes to everyday perceiving by focusing on the following three objectives: (1) Exploring vision's influence on the perception of "hybrid" sounds mixing characteristics of percussive and artificial sounds. (2) Exploring whether percussive sounds are easier to process (one explanation for their propensity to integrate with visual information) by examining reaction times for detecting target intervals amidst a stream of percussive vs. artificial sounds. (3) Testing participants on their ability to intentionally associate sounds with objects, exploring whether performance is better for percussive rather than artificial sounds. Together, these experiments will contribute to our basic understanding of auditory perception by documenting the extent to which previous research using artificial sounds generalizes to the perception of real world auditory events. This work will also be relevant to researchers in other fields. For example, it will be useful to brain scientists investigating the biological basis of multi-sensory perception, an area of neuroscience research that has seen a great deal of growth in recent years. Additionally, our third objective focused on remembering associations between sounds and objects could prove helpful in designing effective auditory human-computer-interfaces, which require users to remember associations between sounds and their intended meaning.