Multi-dimensional and multi-modal integration in perception of vocal music

点击翻译按钮获取中文摘要

Auditory researchers have long known that lip-reading is a normal part of everyday speech and that the integration of audio and visual channels determines our perception of the basic units of speech (i.e., phonemes). For example, if someone is presented with an auditory /ba/ and a visual /ga/, they will generally report an illusory percept of /da/. Similar evidence for audio-visual integration in the perception of vocal music has recently been found by the applicant. This evidence comes from experiments with stimuli that are highly controlled as well as more naturalistic experiments involving audio-video recordings of great vocal performers such as Judy Garland. Experiments in the first phase of the proposed research will use motion analysis and visually-based judgments of musical stimuli to characterize facial movements that are reliably associated with specific units of production in vocal music. Experiments in the second phase will assess properties and possible mechanisms of integration in vocal music with the aim of developing a preliminary model of multi-dimensional and multi-modal integration. This model would be of significant interest to music science and may be of importance to leading edge issues within cognitive neuroscience. Insofar as vocal music and prosody are based on similar components, this model may also be useful for advancing our scientific understanding of speech prosody (i.e., intonation). Finally, building on the work in the first two phases of this grant, a third phase of experiments will lead to a proof-of-concept for an innovative assistive technology. The technology would use a visual display (e.g., an iPod) to project either a close-up view of a singer's face or a point-light animation that captures the critical time-varying aspects of facial movement. The efficacy of the proposed technology to enhance the experience of vocal music in hearing impaired listeners will be tested.