HCC: Small: Collaborative Research: Gestural and Linguistic Expressivity and Entrainment in Dialogue

这项研究将在两个方面推进会话字符系统（智能虚拟代理）的最新技术。首先，它旨在更好地理解手势和语言在产生个性感知和允许参与者适应正在进行的会话上下文方面的相互作用。其次，它将使用这种理解来构建新的计算模型，用于手势和语言生成，提供对个性感知的细粒度控制，并支持代理适应会话上下文。 本研究的人格模型的理论基础是人格的“大五”模型，该模型由五个正交的个体差异维度组成;顺应研究的理论基础是语言使用的协作理论和交际顺应理论，该理论预测交际行为会因交际对象的特殊性而发生变化。初步工作将形成动作捕捉、视频和音频三种交流的语料库。这将用于研究人类互动过程中的手势夹带，确定基于音频的研究结果是否扩展到手势领域，并增强对手势和个性之间关系的科学理解。这将为建模工作提供信息，建模工作将为基于个性的语言和手势制作构建联合模型。该模型将把一项关于外倾性手势生成的试点研究扩展到三个大五特质，并将其与基于个性的语言生成相结合。实验阶段将验证这些模型，并研究运动和语言的相互作用。该研究还将研究适应的作用。要回答的问题包括：（1）人们是否手势夹带与计算机，或确实产生任何手势，而与计算机进行通信，（2）是否计算机？手势诱导促进了与用声音诱导观察到的相似水平的联系，以及（3）在交互过程中改变手势诱导是否比从交互开始就对准手势更有力。角色系统对于从虚拟世界到辅导系统的一系列应用变得越来越重要。越来越多的证据表明，通过这些角色呈现个性的方式，以及它们模仿预期人类行为（如夹带）的程度，对使用它们的应用程序的有效性有直接影响。例如，它将对学生的学习产生直接影响。随着这些应用在社会中，特别是在儿童中变得越来越普遍，重要的是要能够充分利用它们的好处，并避免意想不到的负面后果。这涉及到计算模型的进步，允许代理反映给定的个性并适应人类用户，以及对个性和适应在有效的人类-代理交互中的作用的更深入理解。

This research will advance the state of the art in conversational character systems (Intelligent Virtual Agents) in two ways. First, it seeks to better understand the interplay of gesture and language in both generating the perception of personality and allowing participants to adapt to the ongoing conversational context. Second, it will use this understanding to build novel computational models for gesture and language generation that provide fine grained control over the perception of personality and support agent adaptation in response to the conversational context. The theoretical basis for the personality-based modeling in this project is the well-established "Big Five" model of personality, which consists of five orthogonal dimensions of individual variation.The work on adaptation will be couched in the collaborative theory of language use and communication accommodation theory, which predict that communicative behavior varies based on partner specificity. Initial work will form a motion capture, video and audio corpus of three kinds of exchanges. This will be used to both study gestural entrainment during human interactions, determining if audio-based findings extend to the gestural domain, and to enhance scientific understanding of the relationship between gesture and personality. This will inform the modeling work which will build a joint model for personality-based language and gesture production. The model will extend a pilot study on gesture generation for extraversion to three Big Five traits and integrate it with personality-based language generation. An experimental stage will validate these models and study the interactions of movement and language. The research will also study the role of adaptation. Questions to be answered include: (1) whether people gesturally entrain with computers, or indeed produce any gestures while communicating with a computer, (2) whether computers? gestural entrainment promotes similar levels of affiliation as observed with vocal entrainment, and (3) whether changing gestural entrainment over the course of an interaction is more powerful than aligning gestures from the outset of an interaction. Character systems are becoming increasingly important for a range of applications, from virtual worlds to tutoring systems. There is growing evidence that the way personality is presented through these characters, and how well they mimic expected human behavior like entrainment, has a direct impact on the effectiveness of the applications in which they are used. For example, it will have a direct impact on student learning. As these applications become more ubiquitous in society, particularly among children, it is important to be able to harness their full benefit, and indeed, avoid unintended negative consequences. This involves both advances in computational models that allow an agent to reflect a given personality and adapt to a human user, and also a deeper understanding of the role of personality and adaptation in effective human-agent interactions.