From data and theory to computational models of more effective virtual human gestures

隐喻在人们思考和传达抽象概念的方式中起着核心作用。事实上，“传达一个想法”是一种隐喻形式，就好像一个想法是一个可以被物理传递的实物。我们谈到向某人传达想法，给某人一个想法，拥有一个想法(拥有)或掌握一个想法。研究已经广泛地记录了大量的隐喻在我们的语言中的使用，更根本的是在我们的思维过程中。这些隐喻也反映在我们的手势中。具体地说，隐喻手势为这些隐喻提供了一种物理表现形式。演讲者模仿拿着一个想象中的物体来暗示一个想法。这个想象中的物体的大小可以表明这个想法的重要性。演讲者也使用物理空间来组织他们的话语。例如，想法之间的对比差异，这个和那个，可以通过将想法称为位于不同位置的物理对象的手势来传达，因此位置的差异意味着想法的差异。空间中的特定位置也可以从潜在的隐喻中获得意义，例如，对时间的一个常见隐喻是，时间中的事件是空间中沿着一条线的点，因此在某些文化中，位于左侧的事件被理解为过去，而位于右侧的事件被理解为未来。手势会影响理解和信息回忆：当指令伴随手势时，听者更有可能学习和记住它(Cook&Goldin-Meadow，2006)。它们还通过影响对个人和社会特征的推断，影响听者对说话人的人际态度(Goldin-Meadow&Alibali，2013)。特别是，使用隐喻手势的人被认为比不使用隐喻手势的人更有说服力和能力(Maricchiolo等人。2009年)。隐喻和隐喻手势通过将视觉特征与抽象概念相关联并将其放置在空间中，使人们能够在抽象概念上使用空间推理机制，从而促进理解(Beaudoin-Ryan&Goldin-Meadow，2014；卡布里斯，2011；Kendon，2004；McNeill，2005)。本研究的重点是隐喻与手势之间关系的计算机建模。除了加深对抽象概念、隐喻和肢体手势之间关系的理解外，这项工作的主要目标是改善人机交互(HCI)。人机界面研究探索了如何让机器变得更像人类。具体地说，虚拟人，即具有拟人化特征和行为的自治代理，使用与人类相同的语言和非语言行为，让用户进行面对面的交互。虚拟人技术已被证明在广泛的应用中是有益的，包括教医生如何向患者传达坏消息的虚拟患者，激励学生学习的家教，老年人的助手，以及寻求行为改变的说服性健康干预。这项提议是一项多学科的努力，旨在通过计算对隐喻手势进行建模，并系统地评估它们在人与虚拟人交互中的效果。我们还认为，这项工作将对能够与人互动的社交机器人的设计产生重大影响。

Metaphors play a central role in how people think about and convey abstract ideas. In fact, 'conveying an idea' is a form of metaphor, as if an idea is a physical object that can be physically transferred. We speak of getting ideas across to someone, of giving someone an idea, of possessing ideas (having) or grasping an idea. Research has extensively documented a wide array of metaphor use in our language and more fundamentally in our thought processesSuch metaphors are also reflected in our gesturing. Specifically, metaphoric gestures provide a physical manifestation for these metaphors. Speakers mimic holding an imaginary object to suggest an idea. The importance of the idea may be indicated by the size of this imaginary object. Speakers also use the physical space to structure their discourse. For example, contrasting differences between ideas, this versus that, may be conveyed by gestures that refer to the ideas as physical objects located in distinct locations, thus differences in location imply differences in the ideas. Particular locations in space can also derive meaning from the underlying metaphor, For example, a common metaphor for time is that events in time are points in space along a line so that, in some cultures, events located on the left are understood as being in the past while events on the right are in the future. Gestures are known to influence understanding and information recall: listeners are more likely to learn and remember an instruction when it is accompanied by gesture (Cook & Goldin-Meadow, 2006). They also impact interpersonal attitudes of listeners towards speakers, by influencing inferences about personal and social features (Goldin-Meadow & Alibali, 2013). In particular, speakers who use metaphoric gestures are judged more persuasive and competent than those who do not (Maricchiolo et al. 2009). By associating visual features to abstract concepts and laying them out in space, metaphors and metaphoric gestures enable the use of spatial reasoning mechanisms on abstract concepts and facilitate comprehension (Beaudoin- Ryan & Goldin-Meadow, 2014; Calbris, 2011; Kendon, 2004; McNeill, 2005). The focus of this research is the computational modeling of the relation between metaphor and gesture. Beyond fostering a deeper understanding of the relation between abstract concepts, metaphor and physical gesture, the main goal of this work is to improve Human-Computer Interaction (HCI). HCI research has explored how we can make the machine more human-like. Specifically, virtual humans, autonomous agents with anthropomorphic features and behaviors engage users in face-to-face interactions, using the same verbal and nonverbal behaviors as humans. Virtual human technology has proven beneficial in a wide range of applications, including virtual patients that teach doctors how to break bad news to patients, tutors that motivate students to learn, assistants to the elders and persuasive health interventions seeking behavioral change.This proposal is a multidisciplinary effort to computationally model metaphoric gestures and systematically evaluate their effect in human-virtual human interaction. We also see this work as having a significant impact on the design of social robots that can interact with people.