Spatial Interpretation of Pointing Gestures

当我们想把另一个人的注意力引向一个特定的位置或物体时，我们经常使用指向手势。指向手势在人类交流中无处不在，在人机交互和人机交互中越来越重要。当难以用语言表示物体时，它们通过补充或替代语言来促进互动（例如，指向夜空中的一颗特定的星星）。为了理解这种指向手势，观察者必须确定——除其他事项外——指针想要指示的空间位置。有趣的是，人们如何从另一个人的指向手势中提取被指向的位置还不是很清楚。在最近的工作中，我们通过确定阻碍先前研究进展的两个重要因素来推进我们的理解。首先，我们展示了需要精确控制指针的姿势。相比之下，像以前的研究中经常做的那样，仅仅测量（naïve）指针所指向的位置与观察者所确定的位置之间的差异是不够的。其次，尽管观察者可能试图从指针的手臂和手指推断出一条线，但他们的推断实际上是非线性的。我们用一个模型解释了非线性，该模型将外推解释为线性外推和关于可能参考位置的先验假设的贝叶斯最优积分。这个模型揭示了指针的姿势和观察者对指针行为的空间解释之间的高度合法的关系。在这个项目中，我们系统地研究了在高度控制的虚拟现实环境中，重要变量如何影响指向手势的空间解释。我们期望这能够准确地预测给定的指向手势将如何被特定的观察者解释，以及哪种指向手势最适合挑选出特定的物体或位置。将进行九个实验来解决以下问题。第一个问题是相关的空间维度如何影响指向手势的解释。第二个问题是关于观察者的观点如何影响对指向手势的解释。作为第三个问题，我们解决了如何解释非规范的指向手势，其中手臂和手指没有对齐。作为第四个问题，我们讨论了指向手势的空间解释如何导致识别一个物体，以及物体属性（大小、相关性）如何影响这一过程。第五，我们测试了在高度控制的虚拟现实环境中识别的过程是否也反映在人类之间更自然的交流中。研究结果可用于改善现实生活、虚拟或机器人系统中的具身交互，并为人类感知的基本机制提供一个测试平台。

When we want to direct the attention of another person to a specific location or object, we often use pointing gestures. Pointing gestures are ubiquitously in the communication between humans and get increasingly relevant in human-computer and human-robot interaction. They facilitate interactions by complementing or replacing speech when it is difficult to indicate objects verbally (e.g. pointing at a specific star in the night sky). To understand such pointing gestures, an observer has to determine - among other things - which spatial location a pointer wants to indicate. Interestingly, how people extract the pointed-at position from another person's pointing gesture is not well understood. In recent work, we advanced our understanding by identifying two important factors that thwarted progress in previous studies. First, we showed that the pointer's posture needs to be precisely controlled. By contrast, it is insufficient to measure only the discrepancy between the position at which a (naïve) pointer points and the position identified by an observer, as has often been done in previous studies. Second, albeit observers may attempt to extrapolate a line from the pointer's arm and finger, their extrapolations are actually non-linear. We explained the non-linearity with a model that construes the extrapolation as the Bayesian-optimal integration of linear extrapolation and prior assumptions about likely referent positions. This model revealed a highly lawful relationship between a pointer's posture and an observer's spatial interpretation of the pointing act.With the proposed project, we systematically examine how important variables affect the spatial interpretation of pointing gestures in a highly controlled virtual reality environment. We expect that this enables to predict accurately how a given pointing gestures will be interpreted by a specific observer and which pointing gestures is best suited to single out a specific object or location.Nine experiments will be conducted that address the following questions. The first question is how the relevant spatial dimensions affect how pointing gestures are interpreted. The second question addresses how the observer's view-point affects the interpretation of pointing gestures. As a third question, we address how non-canonical pointing gestures are interpreted, in which the arm and finger are not aligned. As a forth question, we address how the spatial interpretation of a pointing gesture leads to the identification of one object and how object properties (size, relevance) affect this process. Fifth, we test whether the processes identified in highly controlled virtual reality settings are also reflected in more natural communication between humans. The results may be applied to improve embodied interaction in real-life, virtual or robotic systems, and provide a testbed for basic mechanisms of human perception.