Representation of Pointing Uncertainty for the Integration of Pointing Gestures and Speech

指点手势在日常生活中随处可见。通常，指向是用来将他人的注意力引向环境中遥远的物体，特别是当这些物体很难用语言描述时(例如，夜空中的一颗星星或糕点店里的一块特定巧克力)。尽管指点可能会有帮助，但指针并不期望仅靠指点手势就能建立一个共同的注意力焦点。因此，指点通常与言语相辅相成。要选择有意义的口头描述，指针需要对与其手势相关的不确定性有所了解。也就是说，如果一个指示者认为他的手势可能会将另一个人的注意力准确地引向所指对象，那么所给出的口头描述(如果有的话)比他期望该点只将注意力吸引到所指对象的粗略附近时要少。同样地，指点手势的观察者代表了他们自己的知觉不确定性，以知道在哪个区域寻找参照物或判断他们猜测的置信度。总而言之，与指点手势和言语的有效交流需要指示者和观察者侧面的知觉不确定性的表征。然而，关于指针假设通过指向传递的具体信息，观察者假设可以从点推断出来，以及它们与观察者从指向手势实际推断的信息之间的关系，我们知之甚少。为了检验不确定性的表征，我们计划使用视觉搜索任务，在视觉搜索任务中，只有指示器知道目标的身份，指示器通过指向和语音将其传达给观察者。眼球跟踪数据用于导出不确定性的表示，因为指示器和观察者需要在视觉上处理(假定的)对象周围的不确定性区域，以提供适当的口头描述或在搜索对象时。此外，不确定性的表示是从指示器口头描述的特殊性和观察者对其猜测的置信度所需的特殊性程度得出的。工作方案包括在真实和虚拟环境中的实验。WP 1旨在建立基本范式。WP 2讨论了不确定性的表征是否反映了已知的影响指向知觉的因素。WP 3阐述了不确定性的表征如何适应知觉表现。WP 1-3的结果被用来扩展我们的指点知觉的正式模型，以包括不确定性的表示。WP 4评估了通过在真实和虚拟环境中明确地表达对不确定性的误解来改进基于指点的交流的方法。综上所述，我们期望该项目对指点知觉的不确定性及其对手势和言语整合的影响的表征提供新的见解。

Pointing gestures are ubiquitous in every-day life. Typically, pointing is used to direct the attention of others to distant objects in the environment, especially when the objects are difficult to describe verbally (e.g., a star in the night sky or a specific chocolate at a pastry shop). Helpful as pointing may be, pointers do not expect that the pointing gesture alone establishes a joint focus of attention. Hence, pointing is typically complemented by speech. To choose a meaningful verbal description, a pointer needs some notion about the uncertainty associated with his gesture. That is, if a pointer assumes that his gesture may direct another person’s attention exactly to the referent, fewer verbal descriptions (if any) are given than when he expects the point to only draw attention to the rough vicinity of the referent. Likewise, observers of pointing gestures represent their own perceptual uncertainty to know in which area to look for the referent or to judge the confidence of their guess.In summary, effective communication with pointing gestures and speech requires representations of perceptual uncertainty on the sides of the pointers and observers. However, little is known about the specific information pointers assume to convey with pointing, observer assume can be inferred from a point, and how they relate to the information that observers factually infer from pointing gestures. The objective of the present proposal is to examine such representations of uncertainty for the typical case of pointing to small and distant objects.To examine representations of uncertainty, we plan to use visual search tasks, in which the identity of a target is only known to a pointer, who communicates it to an observer with pointing and speech. Eye-tracking data is used to derive representations of uncertainty as pointers and observers need to visually process an area of uncertainty around the (assumed) referent to provide adequate verbal descriptions or when searching for the referent. Additionally, representations of uncertainty are derived from the specificity of the pointer’s verbal descriptions and the level of specificity needed by the observer to be confidence about their guess.The work programme includes experiments in real and virtual settings. WP 1 aims at establishing the basic paradigm. WP 2 addresses whether representations of uncertainty reflect factors that are known to affect pointing perception. WP 3 addresses how representations of uncertainty may be adapted to perceptual performance. The results of WP 1 – 3 are used to extent our formal model of pointing perception to include representations of uncertainty. WP 4 evaluates methods to improve pointing-based communication by making misconceptions about uncertainty explicit in real and virtual settings. In summary, we expect that the project provides new insights into representations of the uncertainty of pointing perception and their effect on the integration of gestures and speech.