EAGER: Investigating the Role of Discourse Context in Speech-Driven Facial Animations

这一早期概念探索性研究分析了话语和对话语境在会话代理(CA)(即与用户交互的虚拟代理)生成可信的、类似人类的行为中所起的作用。CAS的目标是通过展示类似人类的行为来吸引用户，不仅通过语音，还通过面部手势。语音是驱动面部行为的一种有用的方式。口头语言承载的重要信息超出了CA引擎应该利用的口头信息。在语音驱动的动画中，一个挑战是生成响应于话语上下文的行为。该建议提出了一种自上而下的方法来探索在语音驱动的面部手势建模中考虑上下文信息的重要性。该项目从基于动态贝叶斯网络的语音驱动模型开始，该模型不捕捉特定的话语上下文，只对声学特征的属性做出反应。然后，研究考虑了特定于话语的模式，在这些模式中，手势的意图是已知的。这项研究将一个特定的受控领域定义为试验床，记录了多次人类互动。在特定语篇功能的约束下，训练出相似的语音驱动模型。这项研究评估了所产生的面部手势在感知的自然性、适当性和融洽方面的差异。本研究探讨了哪些话语层面影响面部动画模型，哪些更具领域特异性或独立性。通过结合固有的话语信息，所提出的模型产生响应对话功能的行为，解决了语音驱动的面部动画的局限性之一。这些发现对各种医疗保健应用都有长期影响，例如帮助听力受损的个人和向自闭症儿童传授社交技能。同样，依赖于话语的言语驱动模式可以在更好的辅导系统中发挥关键作用，该系统展示了与学生交流和互动的类似人类的行为。

This EArly-concept Grants for Exploratory Research analyzes the role of discourse and dialog context in the generation of believable, human-like behaviors for conversational agent (CA), i.e., a virtual agent that interacts with a user. CAs aim to engage the users by displaying human-like behaviors not only through speech by also through facial gestures. One useful modality to drive facial behaviors is speech. Spoken language carries important information beyond the verbal message that a CA engine should capitalize on. A challenge in speech-driven animation is to generate behaviors that respond to the discourse context. This proposal presents a top-down approach to explore the importance of considering contextual information in the modeling of speech-driven facial gestures. The project starts with speech-driven models, based on dynamic Bayesian networks, which do not capture the specific discourse context, responding only to the properties of the acoustic features. Then, the study considers discourse-specific models in which the intent of the gestures is known. The study defines a specific, controlled domain as testbed, recording multiple human interactions. Similar speech-driven models are trained constrained by the specific discourse function. The study evaluates the differences in the perceived naturalness, appropriateness and rapport of generated facial gestures. The study explores which discourse aspects affect the facial animation models, and which are more domain specific or independent. By incorporating the intrinsic discourse information, the proposed models generate behaviors that respond to conversational functions, addressing one of the limitations in speech-driven facial animations. The findings have a longterm impact in variety of health care applications, such as helping hearing impaired individuals and teaching social skills to autistic children. Likewise, discourse-dependent speech-driven models can play a key role in better tutoring systems that display human-like behaviors to communicate and engage with the students.