HCC: Small: Collaborative Research: Gestural and Linguistic Expressivity and Entrainment in Dialogue

这项研究将从两个方面推动对话角色系统（智能虚拟代理）的发展。首先，它试图更好地理解手势和语言在产生个性感知和允许参与者适应正在进行的对话环境中的相互作用。其次，它将利用这种理解为手势和语言生成建立新的计算模型，提供对个性感知的细粒度控制，并支持智能体根据会话上下文进行适应。本项目基于人格的建模的理论基础是完善的人格“大五”模型，该模型由个体变异的五个正交维度组成。语言使用的合作理论和交际适应理论预测交际行为会根据伴侣的特异性而变化。工作初期将形成动作捕捉、视频和音频语料库三种交流方式。这将用于研究人类互动过程中的手势娱乐，确定基于音频的研究结果是否延伸到手势领域，并增强对手势和个性之间关系的科学理解。这将告知建模工作，这将建立一个基于个性的语言和手势生产的联合模型。该模型将扩展对外向性手势生成的试点研究，将其扩展到三大五大特征，并将其与基于个性的语言生成相结合。实验阶段将验证这些模型，并研究动作和语言的相互作用。该研究还将研究适应的作用。需要回答的问题包括：(1)人们在与电脑交流时是否会产生手势，或者确实会产生任何手势；(2)电脑是否会产生手势？(3)在互动过程中改变手势是否比在互动一开始就调整手势更有效。从虚拟世界到辅导系统，角色系统在一系列应用中变得越来越重要。越来越多的证据表明，通过这些角色呈现个性的方式，以及他们模仿预期人类行为（如娱乐）的程度，对使用这些角色的应用程序的有效性有直接影响。例如，它会对学生的学习产生直接影响。随着这些应用在社会中变得越来越普遍，特别是在儿童中，能够充分利用它们的好处，并确实避免意外的负面后果是很重要的。这既涉及计算模型的进步，允许代理反映给定的个性并适应人类用户，也涉及对个性和适应在有效的人-代理交互中的作用的更深层次的理解。

This research will advance the state of the art in conversational character systems (Intelligent Virtual Agents) in two ways. First, it seeks to better understand the interplay of gesture and language in both generating the perception of personality and allowing participants to adapt to the ongoing conversational context. Second, it will use this understanding to build novel computational models for gesture and language generation that provide fine grained control over the perception of personality and support agent adaptation in response to the conversational context. The theoretical basis for the personality-based modeling in this project is the well-established "Big Five" model of personality, which consists of five orthogonal dimensions of individual variation.The work on adaptation will be couched in the collaborative theory of language use and communication accommodation theory, which predict that communicative behavior varies based on partner specificity. Initial work will form a motion capture, video and audio corpus of three kinds of exchanges. This will be used to both study gestural entrainment during human interactions, determining if audio-based findings extend to the gestural domain, and to enhance scientific understanding of the relationship between gesture and personality. This will inform the modeling work which will build a joint model for personality-based language and gesture production. The model will extend a pilot study on gesture generation for extraversion to three Big Five traits and integrate it with personality-based language generation. An experimental stage will validate these models and study the interactions of movement and language. The research will also study the role of adaptation. Questions to be answered include: (1) whether people gesturally entrain with computers, or indeed produce any gestures while communicating with a computer, (2) whether computers? gestural entrainment promotes similar levels of affiliation as observed with vocal entrainment, and (3) whether changing gestural entrainment over the course of an interaction is more powerful than aligning gestures from the outset of an interaction. Character systems are becoming increasingly important for a range of applications, from virtual worlds to tutoring systems. There is growing evidence that the way personality is presented through these characters, and how well they mimic expected human behavior like entrainment, has a direct impact on the effectiveness of the applications in which they are used. For example, it will have a direct impact on student learning. As these applications become more ubiquitous in society, particularly among children, it is important to be able to harness their full benefit, and indeed, avoid unintended negative consequences. This involves both advances in computational models that allow an agent to reflect a given personality and adapt to a human user, and also a deeper understanding of the role of personality and adaptation in effective human-agent interactions.