EAGER: Investigating the Role of Discourse Context in Speech-Driven Facial Animations

EAGER：研究话语上下文在语音驱动的面部动画中的作用

• 批准号: 1352950
• 负责人: Yang Liu
• 金额: $ 8.26万
• 依托单位: University of Texas at Dallas
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2013
• 资助国家: 美国
• 起止时间: 2013-09-01 至 2016-02-29
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1352950&HistoricalAwards=false
• 关键词: EAGER; Investigating; Role; Discourse; Context

This EArly-concept Grants for Exploratory Research analyzes the role of discourse and dialog context in the generation of believable, human-like behaviors for conversational agent (CA), i.e., a virtual agent that interacts with a user. CAs aim to engage the users by displaying human-like behaviors not only through speech by also through facial gestures. One useful modality to drive facial behaviors is speech. Spoken language carries important information beyond the verbal message that a CA engine should capitalize on. A challenge in speech-driven animation is to generate behaviors that respond to the discourse context. This proposal presents a top-down approach to explore the importance of considering contextual information in the modeling of speech-driven facial gestures. The project starts with speech-driven models, based on dynamic Bayesian networks, which do not capture the specific discourse context, responding only to the properties of the acoustic features. Then, the study considers discourse-specific models in which the intent of the gestures is known. The study defines a specific, controlled domain as testbed, recording multiple human interactions. Similar speech-driven models are trained constrained by the specific discourse function. The study evaluates the differences in the perceived naturalness, appropriateness and rapport of generated facial gestures. The study explores which discourse aspects affect the facial animation models, and which are more domain specific or independent. By incorporating the intrinsic discourse information, the proposed models generate behaviors that respond to conversational functions, addressing one of the limitations in speech-driven facial animations. The findings have a longterm impact in variety of health care applications, such as helping hearing impaired individuals and teaching social skills to autistic children. Likewise, discourse-dependent speech-driven models can play a key role in better tutoring systems that display human-like behaviors to communicate and engage with the students.

这个早期概念的探索性研究赠款分析了话语和对话语境在会话主体（CA）产生可信的、类似人类的行为中的作用，即，与用户交互的虚拟代理。CA的目标是通过显示类似人类的行为来吸引用户，不仅通过语音，还通过面部手势。驱动面部行为的一种有用的方式是语音。口语承载着CA引擎应该利用的语言信息之外的重要信息。语音驱动动画的挑战是生成响应话语上下文的行为。该建议提出了一种自上而下的方法来探索考虑上下文信息的重要性，在语音驱动的面部姿态建模。该项目从基于动态贝叶斯网络的语音驱动模型开始，该模型不捕获特定的话语上下文，仅响应声学特征的属性。然后，该研究考虑特定于话语的模型，其中手势的意图是已知的。该研究定义了一个特定的，受控的域作为测试床，记录多个人类互动。类似的语音驱动模型的训练受到特定话语功能的约束。该研究评估了在感知的自然性，适当性和融洽的生成的面部表情的差异。本研究探讨了哪些话语方面影响面部动画模型，哪些话语方面更具有领域特异性或独立性。通过将内在的话语信息，所提出的模型生成的行为，响应会话功能，解决语音驱动的面部动画的局限性之一。这些发现对各种医疗保健应用产生了长期影响，例如帮助听力受损的人和向自闭症儿童教授社交技能。同样，依赖于话语的语音驱动模型可以在更好的辅导系统中发挥关键作用，这些系统可以显示类似人类的行为来与学生进行交流和互动。

项目成果

Meaningful head movements driven by emotional synthetic speech

由情感合成语音驱动的有意义的头部运动

• DOI: 10.1016/j.specom.2017.07.004
• 发表时间: 2017
• 期刊: Speech Communication
• 影响因子: 3.2
• 作者: Sadoughi, Najmeh;Liu, Yang;Busso, Carlos
• 通讯作者: Busso, Carlos