RI: Small: Integrative, Semantic-Aware, Speech-Driven Models for Believable Conversational Agents with Meaningful Behaviors

这个项目将分析，建模和合成人类行为，以创建一个可信的会话代理（CA）。CA是一个与用户交互的虚拟代理，不仅通过语音，还通过面部表情和头部运动来显示类似人类的行为。复制或表示人类行为包括生成与语音同步的手势，在消息中传达适当的含义，并对用户显示的行为做出响应。一种吸引人的方法来合成类人行为是使用数据驱动的方法，它有可能捕捉行为的自然变化。对语音和手势之间的依赖关系进行建模，可以深入了解语言和非语言交流，这是自然人类互动过程中使用的生产和协调机制的基础。CA可以用于各种医疗保健应用，例如帮助听力受损的人和向自闭症儿童教授社交技能。显示类似人类行为的辅导系统进行沟通并承认积极倾听，将更好地与学生互动，帮助他们学习。该项目为研究生和本科生的跨学科培训提供了肥沃的土壤。这些模型将与辅助代理进行评估（CA或嵌入式机器人）与UT达拉斯学生互动，作为一个平台，接触来自所有专业的学生，特别是女性和代表性不足的少数民族。该项目将采取综合，跨学科的方法，通过探索语音，头部运动和面部表情之间的内在关系，受口语重要方面的限制。计划中的研究以综合的方式利用深度学习领域的一些最新发展，将声学特征和语义语言结构结合在一起，构建能够解释各种面部和头部运动之间相关性的模型。语音驱动的方法将以一种基于规则的方法不容易实现的方式捕获人类行为的可变性。对话行为和情感将被推断并用于约束语音驱动模型，捕获高级会话功能和面部手势之间的关系。该项目将提供新颖的、有原则的方法来生成由合成语音驱动的行为，在只有文本的情况下开辟新的应用领域。该方法将捕获合成语音中的声学变化，同时保持手势和语音之间的时间依赖性。该项目还将探索通过显示由我们的数据驱动框架生成的精心设计的手势来修改用户行为的方案。通过跟踪用户的行为，系统将提供适当的响应，关闭交互中的循环。

This project will analyze, model and synthesize human behaviors to create a believable Conversational Agent (CA). A CA is a virtual agent that interacts with a user, displaying human-like behaviors not only through speech but also through facial expressions and head movements. Replicating or representing human behavior includes generating gestures that are synchronized with speech, convey appropriate meaning in the message, and respond to the behaviors displayed by the user. An appealing approach to synthesize human-like behaviors is the use of data-driven methods, which have the potential of capturing naturalistic variations of the behaviors. Modeling the dependencies between speech and gestures brings insights about verbal and nonverbal communication, underlying the production and coordination mechanisms used during natural human interactions. CAs can be used in a variety of health care applications, such as helping hearing impaired individuals and teaching social skills to autistic children. Tutoring systems that display human-like behaviors to communicate and acknowledge active listening will engage better with the students, helping them in their learning. The project promises a fertile ground for interdisciplinary training of graduate and undergraduate students. The models will be evaluated with an assistive agent (CA or embodied robot) interacting with UT Dallas students, serving as a platform to reach out students from all majors, especially woman and underrepresented minorities.The project will take an integrative, cross-disciplinary approach to generate believable and meaningful behaviors by exploring the intrinsic relation between speech, head motion, and facial expressions, constrained by important aspects of spoken language. The planned research leverages some of the latest developments in the field of deep learning in an integrative fashion, pulling together acoustic features and semantic language structure, to build models that are able to account for the correlation between various facial and head movements. The speech-driven approach will capture the variability of human behavior in a manner that is not easily possible with rule-based approaches. Dialog acts and emotions will be inferred and used to constrain the speech driven models, capturing the relation between high-level conversational functions and facial gestures. The project will offer novel, principled methods to generate behaviors driven by synthesized speech, opening new application domain when only text is available. The approach will capture the acoustic variability in synthesized speech, while maintaining the temporal dependency between gestures and speech. The project will also explore schemes to modify the behaviors of the user by displaying carefully designed gestures generated with our data-driven framework. By tracking the behaviors of the user, the system will provide appropriate responses, closing the loop in the interaction.