The Temporal Organization of Prosody in Multimodal Speech and Co-speech Gestures

多模态语音和协同语音手势中韵律的时间组织

• 批准号: 2105410
• 负责人: Yoonjeong Lee
• 金额: $ 13.8万
• 依托单位: Lee, Yoonjeong
• 依托单位国家: 美国
• 项目类别: Fellowship Award
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2105410&HistoricalAwards=false
• 关键词: Temporal; Organization; Prosody; Multimodal; Speech

This award was provided as part of NSF’s Social, Behavioral and Economic Sciences Postdoctoral Research Fellowships (SPRF) program. The goal of the SPRF program is to prepare promising, early career doctoral-level scientists for scientific careers in academia, industry or private sector, and government. SPRF awards involve two years of training under the sponsorship of established scientists and encourage Postdoctoral Fellows to perform independent research. NSF seeks to promote the participation of scientists from all segments of the scientific community, including those from underrepresented groups, in its research programs and activities; the postdoctoral period is considered to be an important level of professional development in attaining this goal. Each Postdoctoral Fellow must address important scientific questions that advance their respective disciplinary fields. Under the sponsorship of Dr. Jelena Krivokapić at the University of Michigan, this postdoctoral fellowship award supports an early career scientist examining the nature of coordination between multimodal speech events. In everyday communication, speech and gestures that accompany speech, referred to as co-speech gestures, interact in a principled way. While much of interdisciplinary research has examined the multifaceted nature of verbal and non-verbal behaviors in human communication, our understanding of the multimodal prosodic phenomena remains limited. The goal of the proposed research is to illuminate how language users integrate prosody into different modalities while producing words and phrases. Prosodic structure groups words into larger phrases and expresses prominence (highlighting important or new information). Specifically, this project examines how prosodic information is embodied in co-speech gestures occurring at phrasal edges and under prominence, what coordination patterns of speech and co-speech gestures emerge from the prosodic structure that is specific to a language, and if and how interacting language users exhibit prosodic accommodation in co-speech gestures. By using cutting-edge instruments and analysis tools, multimodal data (pitch, movement of the vocal tract, head, eyebrow, hand) are assessed via concurrent recordings of audio, video, and kinematic information from individual speakers of typologically different languages (English and Korean) participating in various speech tasks including solo and interactive speech activities.The proposed research examines how the speech signal and the accompanying bodily movements produced by a language user are mediated by prosodic structure of the language. This project utilizes cutting-edge techniques to collect the necessary multimodal data and characterizes the temporal relation between co-speech beat gestures, pitch gestures, and consonant and vowel gestures as well as detailing their kinematic properties at phrasal boundaries and under prominence. Further, a series of multimodal experiments is designed for two prosodically different languages—a language with no lexical stress and no pitch accent and a language with stress and marking prominence with pitch accents to reveal the cross-linguistic and/or language-specific aspects of speech and co-speech gesture coordination. This research also conducts a real-time accommodation study designed for pairs of speakers participating in communicative speech tasks, with the broad goal of gaining information about the emergence of prosodic accommodation across modalities. To date, the proposed work is the first to investigate accommodation processes in both speech and co-speech gestures. This work can advance our understanding of the precise nature of the speech and co-speech gesture coordination and of how different modalities are coordinated to embody information grouping and highlighting at the phrasal level, broadening the empirical and theoretical base for models of articulatory organization and linguistic accounts of prosody. The results serve to support the development of theoretical models of prosodic structure of typologically different languages that take into account these different modalities of communication. The proposed work produces a substantial database of the movement signals of multimodal articulators with synchronous audio and video recordings that will be freely distributed to the research and education community. Broadly, this research contributes to knowledge and perspectives for future exploration of multimodality in communication in sign language, education, and engineering and clinical applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该奖项是 NSF 社会、行为和经济科学博士后研究奖学金 (SPRF) 计划的一部分。 SPRF 项目的目标是为学术界、工业界或私营部门以及政府的科学职业培养有前途的早期职业博士级科学家。 SPRF 奖项包括在知名科学家的赞助下进行两年的培训，并鼓励博士后研究员进行独立研究。 NSF 致力于促进科学界各个领域的科学家（包括来自代表性不足群体的科学家）参与其研究计划和活动；博士后时期被认为是实现这一目标的重要专业发展阶段。每个博士后研究员必须解决推动各自学科领域发展的重要科学问题。在密歇根大学 Jelena Krivokapić 博士的赞助下，该博士后奖学金资助一位早期职业科学家研究多模态语音事件之间协调的本质。在日常交流中，言语和伴随言语的手势（称为共同言语手势）以有原则的方式相互作用。尽管许多跨学科研究已经检验了人类交流中言语和非言语行为的多方面性质，但我们对多模态韵律现象的理解仍然有限。拟议研究的目标是阐明语言用户在产生单词和短语时如何将韵律整合到不同的模态中。韵律结构将单词分组为更大的短语并表达重要性（突出显示重要或新信息）。具体来说，该项目研究韵律信息如何体现在出现在短语边缘和突出部分的协同语音手势中，语音和协同语音手势的协调模式从特定于语言的韵律结构中出现，以及交互语言用户是否以及如何在协同语音手势中表现出韵律适应。通过使用尖端仪器和分析工具，通过同时记录来自不同类型语言（英语和韩语）的个体说话者的音频、视频和运动学信息来评估多模态数据（音调、声道运动、头部、眉毛、手），这些信息来自参与各种语音任务（包括独奏和交互式语音活动）的人。拟议的研究探讨了语言使用者产生的语音信号和伴随的身体运动如何通过 语言的韵律结构。该项目利用尖端技术来收集必要的多模态数据，并表征协同语音节拍手势、音调手势以及辅音和元音手势之间的时间关系，并详细说明它们在短语边界和突出部分的运动学特性。此外，针对两种韵律不同的语言设计了一系列多模态实验——一种没有词汇重音和音调重音的语言，一种有重音并用音调重音标记突出的语言，以揭示语音和共同语音手势协调的跨语言和/或特定语言方面。这项研究还进行了一项实时调节研究，专为参与交际语音任务的说话者而设计，其总体目标是获取有关跨模式的韵律调节出现的信息。迄今为止，拟议的工作是第一个研究语音和共同语音手势中的调节过程的工作。这项工作可以增进我们对语音和共同语音手势协调的精确性质的理解，以及如何协调不同的模态以在短语层面体现信息分组和突出显示，从而拓宽发音组织模型和韵律语言解释的经验和理论基础。研究结果有助于支持不同类型语言的韵律结构理论模型的发展，这些模型考虑到了这些不同的交流方式。拟议的工作产生了一个包含同步音频和视频记录的多模式咬合架运动信号的大量数据库，该数据库将免费分发给研究和教育界。从广义上讲，这项研究为未来探索手语、教育、工程和临床应用中多模态通信的知识和观点做出了贡献。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。