Multi-dimensional and multi-modal integration in perception of vocal music

听觉研究人员早就知道，唇读是日常言语的正常组成部分，听觉和视觉渠道的整合决定了我们对言语基本单位（即音素）的感知。例如，如果有人看到听觉/ba/和视觉/ga/，他们通常会报告对/da/的错觉感知。申请人最近发现了类似的证据，证明在声乐感知中视听整合。这一证据来自于高度受控的刺激实验，以及更自然的实验，包括朱迪·加兰（Judy Garland）等杰出声乐表演者的录音录像。拟议研究的第一阶段的实验将使用动作分析和基于视觉的音乐刺激判断来表征与声乐特定生产单位可靠相关的面部运动。第二阶段的实验将评估声乐整合的特性和可能的机制，目的是建立一个多维和多模态整合的初步模型。这个模型将对音乐科学产生重大的兴趣，并且可能对认知神经科学中的前沿问题很重要。就声乐和韵律基于相似的成分而言，这个模型也可能有助于提高我们对语音韵律（即语调）的科学理解。最后，在本赠款前两个阶段工作的基础上，第三阶段的实验将导致创新辅助技术的概念验证。这项技术将使用视觉显示器（例如iPod）来投射歌手脸部的特写，或者用点光动画来捕捉面部运动的关键时变方面。我们将测试所提出的技术对听觉受损听众提升声乐体验的效果。

Auditory researchers have long known that lip-reading is a normal part of everyday speech and that the integration of audio and visual channels determines our perception of the basic units of speech (i.e., phonemes). For example, if someone is presented with an auditory /ba/ and a visual /ga/, they will generally report an illusory percept of /da/. Similar evidence for audio-visual integration in the perception of vocal music has recently been found by the applicant. This evidence comes from experiments with stimuli that are highly controlled as well as more naturalistic experiments involving audio-video recordings of great vocal performers such as Judy Garland. Experiments in the first phase of the proposed research will use motion analysis and visually-based judgments of musical stimuli to characterize facial movements that are reliably associated with specific units of production in vocal music. Experiments in the second phase will assess properties and possible mechanisms of integration in vocal music with the aim of developing a preliminary model of multi-dimensional and multi-modal integration. This model would be of significant interest to music science and may be of importance to leading edge issues within cognitive neuroscience. Insofar as vocal music and prosody are based on similar components, this model may also be useful for advancing our scientific understanding of speech prosody (i.e., intonation). Finally, building on the work in the first two phases of this grant, a third phase of experiments will lead to a proof-of-concept for an innovative assistive technology. The technology would use a visual display (e.g., an iPod) to project either a close-up view of a singer's face or a point-light animation that captures the critical time-varying aspects of facial movement. The efficacy of the proposed technology to enhance the experience of vocal music in hearing impaired listeners will be tested.