The Temporal Organization of Prosody in Multimodal Speech and Co-speech Gestures

该奖项是美国国家科学基金会社会、行为和经济科学博士后研究奖学金（SPRF）计划的一部分。SPRF计划的目标是为学术界、工业界或私营部门和政府的科学事业准备有前途的早期职业博士级科学家。SPRF奖励包括在知名科学家的赞助下进行为期两年的培训，并鼓励博士后进行独立研究。美国国家科学基金会寻求促进科学界各阶层的科学家，包括那些未被充分代表的群体的科学家，参与其研究项目和活动；博士后阶段被认为是实现这一目标的一个重要的专业发展阶段。每个博士后必须解决各自学科领域的重要科学问题。在密歇根大学的Jelena krivokapiki博士的赞助下，这个博士后奖学金奖支持一位早期职业科学家研究多模态言语事件之间协调的本质。在日常交流中，语言和伴随语言的手势，被称为共同语言手势，以一种有原则的方式相互作用。虽然许多跨学科研究已经考察了人类交际中语言和非语言行为的多面性，但我们对多模态韵律现象的理解仍然有限。本研究的目的是阐明语言使用者在生成单词和短语时如何将韵律整合到不同的形态中。韵律结构将单词组合成更大的短语，并表达突出（突出重要的或新的信息）。具体来说，本项目研究了韵律信息是如何体现在短语边缘和突出位置的同语手势中，语言和同语手势的协调模式是如何从特定于语言的韵律结构中产生的，以及相互作用的语言用户是否以及如何在同语手势中表现出韵律适应性。通过使用先进的仪器和分析工具，通过同时记录不同语言（英语和韩语）的个体说话者的音频、视频和运动学信息来评估多模态数据（音调、声道运动、头部、眉毛、手），这些说话者参与各种语音任务，包括独奏和互动语音活动。本研究旨在探讨语言使用者产生的语音信号和伴随的肢体动作是如何受到语言韵律结构的调节的。本项目利用尖端技术收集必要的多模态数据，表征同音节拍手势、音高手势、辅音和元音手势之间的时间关系，并详细描述它们在短语边界和突出位置下的运动学特性。此外，我们还针对两种韵律不同的语言设计了一系列多模态实验——一种语言没有词汇重音和音高重音，另一种语言有重音和音高重音标记突出，以揭示言语和共言语手势协调的跨语言和/或语言特定方面。本研究还进行了一项实时适应研究，该研究是为参与交际性言语任务的两对说话者设计的，其广泛目标是获得有关不同模式下韵律适应出现的信息。到目前为止，这项工作是第一个研究语音和共同语音手势的适应过程的工作。这项工作可以促进我们对语音和共语音手势协调的确切性质的理解，以及不同的模态如何协调以体现短语层面的信息分组和突出，拓宽了发音组织模型和韵律语言描述的经验和理论基础。研究结果支持了不同类型语言的韵律结构理论模型的发展，该模型考虑了这些不同的交流方式。拟议的工作产生了一个多模态发音器运动信号的大量数据库，其中有同步的音频和视频记录，将免费分发给研究和教育界。从广义上讲，本研究为未来探索手语、教育、工程和临床应用中的多模态交流提供了知识和视角。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This award was provided as part of NSF’s Social, Behavioral and Economic Sciences Postdoctoral Research Fellowships (SPRF) program. The goal of the SPRF program is to prepare promising, early career doctoral-level scientists for scientific careers in academia, industry or private sector, and government. SPRF awards involve two years of training under the sponsorship of established scientists and encourage Postdoctoral Fellows to perform independent research. NSF seeks to promote the participation of scientists from all segments of the scientific community, including those from underrepresented groups, in its research programs and activities; the postdoctoral period is considered to be an important level of professional development in attaining this goal. Each Postdoctoral Fellow must address important scientific questions that advance their respective disciplinary fields. Under the sponsorship of Dr. Jelena Krivokapić at the University of Michigan, this postdoctoral fellowship award supports an early career scientist examining the nature of coordination between multimodal speech events. In everyday communication, speech and gestures that accompany speech, referred to as co-speech gestures, interact in a principled way. While much of interdisciplinary research has examined the multifaceted nature of verbal and non-verbal behaviors in human communication, our understanding of the multimodal prosodic phenomena remains limited. The goal of the proposed research is to illuminate how language users integrate prosody into different modalities while producing words and phrases. Prosodic structure groups words into larger phrases and expresses prominence (highlighting important or new information). Specifically, this project examines how prosodic information is embodied in co-speech gestures occurring at phrasal edges and under prominence, what coordination patterns of speech and co-speech gestures emerge from the prosodic structure that is specific to a language, and if and how interacting language users exhibit prosodic accommodation in co-speech gestures. By using cutting-edge instruments and analysis tools, multimodal data (pitch, movement of the vocal tract, head, eyebrow, hand) are assessed via concurrent recordings of audio, video, and kinematic information from individual speakers of typologically different languages (English and Korean) participating in various speech tasks including solo and interactive speech activities.The proposed research examines how the speech signal and the accompanying bodily movements produced by a language user are mediated by prosodic structure of the language. This project utilizes cutting-edge techniques to collect the necessary multimodal data and characterizes the temporal relation between co-speech beat gestures, pitch gestures, and consonant and vowel gestures as well as detailing their kinematic properties at phrasal boundaries and under prominence. Further, a series of multimodal experiments is designed for two prosodically different languages—a language with no lexical stress and no pitch accent and a language with stress and marking prominence with pitch accents to reveal the cross-linguistic and/or language-specific aspects of speech and co-speech gesture coordination. This research also conducts a real-time accommodation study designed for pairs of speakers participating in communicative speech tasks, with the broad goal of gaining information about the emergence of prosodic accommodation across modalities. To date, the proposed work is the first to investigate accommodation processes in both speech and co-speech gestures. This work can advance our understanding of the precise nature of the speech and co-speech gesture coordination and of how different modalities are coordinated to embody information grouping and highlighting at the phrasal level, broadening the empirical and theoretical base for models of articulatory organization and linguistic accounts of prosody. The results serve to support the development of theoretical models of prosodic structure of typologically different languages that take into account these different modalities of communication. The proposed work produces a substantial database of the movement signals of multimodal articulators with synchronous audio and video recordings that will be freely distributed to the research and education community. Broadly, this research contributes to knowledge and perspectives for future exploration of multimodality in communication in sign language, education, and engineering and clinical applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.