RI: Small: Integrative, Semantic-Aware, Speech-Driven Models for Believable Conversational Agents with Meaningful Behaviors

RI：小型：集成的、语义感知的、语音驱动的模型，用于具有有意义行为的可信会话代理

• 批准号: 1718944
• 负责人: Carlos Busso
• 金额: $ 49.41万
• 依托单位: University of Texas at Dallas
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-09-01 至 2022-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1718944&HistoricalAwards=false
• 关键词: RI; Small; Integrative; Semantic; Aware

This project will analyze, model and synthesize human behaviors to create a believable Conversational Agent (CA). A CA is a virtual agent that interacts with a user, displaying human-like behaviors not only through speech but also through facial expressions and head movements. Replicating or representing human behavior includes generating gestures that are synchronized with speech, convey appropriate meaning in the message, and respond to the behaviors displayed by the user. An appealing approach to synthesize human-like behaviors is the use of data-driven methods, which have the potential of capturing naturalistic variations of the behaviors. Modeling the dependencies between speech and gestures brings insights about verbal and nonverbal communication, underlying the production and coordination mechanisms used during natural human interactions. CAs can be used in a variety of health care applications, such as helping hearing impaired individuals and teaching social skills to autistic children. Tutoring systems that display human-like behaviors to communicate and acknowledge active listening will engage better with the students, helping them in their learning. The project promises a fertile ground for interdisciplinary training of graduate and undergraduate students. The models will be evaluated with an assistive agent (CA or embodied robot) interacting with UT Dallas students, serving as a platform to reach out students from all majors, especially woman and underrepresented minorities.The project will take an integrative, cross-disciplinary approach to generate believable and meaningful behaviors by exploring the intrinsic relation between speech, head motion, and facial expressions, constrained by important aspects of spoken language. The planned research leverages some of the latest developments in the field of deep learning in an integrative fashion, pulling together acoustic features and semantic language structure, to build models that are able to account for the correlation between various facial and head movements. The speech-driven approach will capture the variability of human behavior in a manner that is not easily possible with rule-based approaches. Dialog acts and emotions will be inferred and used to constrain the speech driven models, capturing the relation between high-level conversational functions and facial gestures. The project will offer novel, principled methods to generate behaviors driven by synthesized speech, opening new application domain when only text is available. The approach will capture the acoustic variability in synthesized speech, while maintaining the temporal dependency between gestures and speech. The project will also explore schemes to modify the behaviors of the user by displaying carefully designed gestures generated with our data-driven framework. By tracking the behaviors of the user, the system will provide appropriate responses, closing the loop in the interaction.

这个项目将分析、建模和合成人类行为，以创建一个可信的对话代理(CA)。CA是与用户交互的虚拟代理，不仅通过语音显示类似人类的行为，还通过面部表情和头部运动显示类似人类的行为。复制或表示人类行为包括生成与语音同步的手势，在消息中传达适当的含义，以及响应用户显示的行为。合成类人行为的一个吸引人的方法是使用数据驱动的方法，这种方法有可能捕捉行为的自然变化。对语言和手势之间的依赖关系进行建模可以深入了解语言和非语言交流，这是自然人类互动过程中使用的产生和协调机制的基础。CA可用于各种医疗保健应用，例如帮助听力受损的个人和向自闭症儿童传授社交技能。显示出与人类相似的行为来交流和承认积极倾听的辅导系统将更好地与学生互动，帮助他们学习。该项目为研究生和本科生的跨学科培训提供了肥沃的土壤。这些模型将通过与德克萨斯大学达拉斯分校的学生互动的辅助代理(CA或虚拟机器人)进行评估，作为接触来自所有专业的学生的平台，特别是女性和代表性较低的少数民族。该项目将采取综合、跨学科的方法，通过探索言语、头部运动和面部表情之间的内在联系，产生可信和有意义的行为，这些行为受到口语重要方面的限制。这项计划中的研究以一种综合的方式利用深度学习领域的一些最新发展，将声学特征和语义语言结构结合在一起，以建立能够解释各种面部和头部动作之间的相关性的模型。语音驱动的方法将以一种基于规则的方法不容易实现的方式来捕捉人类行为的可变性。对话行为和情感将被推断并用于约束语音驱动模型，捕捉高级对话功能和面部手势之间的关系。该项目将提供新颖的、原则性的方法来生成由合成语音驱动的行为，在只有文本可用的情况下打开新的应用领域。该方法将捕捉合成语音中的声学变化，同时保持手势和语音之间的时间相关性。该项目还将探索通过显示由我们的数据驱动框架生成的精心设计的手势来修改用户行为的方案。通过跟踪用户的行为，系统将提供适当的响应，从而关闭交互中的循环。

项目成果

AuxFormer: Robust Approach to Audiovisual Emotion Recognition

• DOI: 10.1109/icassp43922.2022.9747157
• 发表时间: 2022-05
• 期刊: ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
• 作者: Lucas Goncalves;C. Busso

Speech-driven Animation with Meaningful Behaviors

• DOI: 10.1016/j.specom.2019.04.005
• 发表时间: 2017-08
• 期刊: Speech Commun.
• 作者: Najmeh Sadoughi;C. Busso

Audiovisual Speech Activity Detection with Advanced Long Short-Term Memory

• DOI: 10.21437/interspeech.2018-2490
• 发表时间: 2018-09
• 作者: Fei Tao;C. Busso

Expressive Speech-Driven Lip Movements with Multitask Learning

• DOI: 10.1109/fg.2018.00066
• 发表时间: 2018-05
• 期刊: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)
• 作者: Najmeh Sadoughi;C. Busso

End-to-End Audiovisual Speech Recognition System With Multitask Learning

• DOI: 10.1109/tmm.2020.2975922
• 发表时间: 2021
• 期刊: IEEE Transactions on Multimedia
• 影响因子: 7.3
• 作者: Fei Tao;C. Busso