Multi-dimensional and multi-modal integration in perception of vocal music

听觉研究人员早就知道唇读是日常言语的正常部分，听觉和视觉通道的整合决定了我们对言语基本单位的感知（即，音素）。例如，如果有人听到一个听觉/ba/和一个视觉/ga/，他们通常会报告一个虚幻的/da/。申请人最近发现了声乐感知中视听整合的类似证据。这一证据来自高度受控的刺激实验，以及更自然的实验，涉及像朱迪加兰这样的伟大声乐表演者的音频视频记录。拟议研究的第一阶段实验将使用运动分析和基于视觉的音乐刺激判断来表征与声乐中特定生产单元可靠相关的面部运动。第二阶段的实验将评估声乐中整合的属性和可能的机制，目的是开发多维和多模态整合的初步模型。这个模型将是音乐科学的重大利益，并可能是认知神经科学的前沿问题的重要性。由于声乐和韵律是基于类似的成分，因此该模型也可能有助于推进我们对语音韵律的科学理解（即，语调）。最后，在前两个阶段工作的基础上，第三阶段的实验将导致创新辅助技术的概念验证。该技术将使用视觉显示器（例如，iPod）来投射歌手面部的特写视图或捕捉面部运动的关键时变方面的点光动画。所提出的技术，以提高听力受损的听众的声乐体验的功效将进行测试。

Auditory researchers have long known that lip-reading is a normal part of everyday speech and that the integration of audio and visual channels determines our perception of the basic units of speech (i.e., phonemes). For example, if someone is presented with an auditory /ba/ and a visual /ga/, they will generally report an illusory percept of /da/. Similar evidence for audio-visual integration in the perception of vocal music has recently been found by the applicant. This evidence comes from experiments with stimuli that are highly controlled as well as more naturalistic experiments involving audio-video recordings of great vocal performers such as Judy Garland. Experiments in the first phase of the proposed research will use motion analysis and visually-based judgments of musical stimuli to characterize facial movements that are reliably associated with specific units of production in vocal music. Experiments in the second phase will assess properties and possible mechanisms of integration in vocal music with the aim of developing a preliminary model of multi-dimensional and multi-modal integration. This model would be of significant interest to music science and may be of importance to leading edge issues within cognitive neuroscience. Insofar as vocal music and prosody are based on similar components, this model may also be useful for advancing our scientific understanding of speech prosody (i.e., intonation). Finally, building on the work in the first two phases of this grant, a third phase of experiments will lead to a proof-of-concept for an innovative assistive technology. The technology would use a visual display (e.g., an iPod) to project either a close-up view of a singer's face or a point-light animation that captures the critical time-varying aspects of facial movement. The efficacy of the proposed technology to enhance the experience of vocal music in hearing impaired listeners will be tested.