Multimodal, Negotiated Interaction in Mobile Scenarios

我们的建议是调查一种允许用户与其环境中的内容和服务交互的替代方法，以便他们所做的动作、动作、手势等以及他们收到的反馈是连续的，用户和系统以流动、动态的方式协商他们的交互。我们认为，恰当的比喻应该是跳舞，而不是目前的指挥与控制比喻。当有人和舞伴跳舞时，有一种控制的柔和潮起潮落；有时一个人领舞，有时另一个人带头，随着他们的舞蹈，这种变化是流畅的。我们建议在用户和计算机之间进行类似的交互，其中有时用户主导，有时根据交互的上下文领导计算机。这与当前大多数方法不同，在大多数方法中，一个代理，无论是人还是计算机，都会先发制人，大多数交互都是由事件驱动的，并以不同程度的僵硬、特定的方式进行。示例场景：探索数字丰富的环境如何使用这种方法的一个示例是在城镇中心行走时获取位置感知信息。你可能会感觉到手机的振动马达上有一个“滴答”，这让你意识到你的环境中有一些可用的信息。你丰富的上下文理解能力将告诉你这个‘滴答’是多么有可能是感兴趣的，如果你无视提示而继续前进，谈判就会当场结束。如果你很好奇，你可以用手机向你周围可能的目标做个手势，然后从其中几个人那里得到回应。如果你进一步感兴趣，你可能会继续与这些潜在的目标互动，可能会从振动触觉转移到音频显示器，通过对环境的积极探索来获取信息，这是我们进化成的自然行为。用户通过直接接触(探索或玩耍)来探索情景中的可能性，能够在可能性的空间中随意移动，在互动中获得越来越多的洞察力。提供的多模式反馈既编码了系统对用户意图(例如，向目标移动)的当前解释，也编码了目标满足用户需求的概率。在经历了振动和音频的组合后，如果信息源和用户的联合动力继续交织在一起，移动设备的显示可能会用于完整的细节。这个例子显示了一种模式的“时间表”，并用一个实际的、具有商业意义的例子说明了谈判过程。

Our proposal is to investigate an alternative means of allowing users to interact with content and services in their environment such that the actions they make, movements, gestures, etc., and feedback they receive are continuous, with the user and system negotiating their interactions in a fluid, dynamic way. We believe the appropriate comparison would be dancing, rather than the current command & control metaphor. When someone dances with a partner there is a soft ebb and flow of control; sometimes one person leads, sometimes the other, this changing fluidly as they dance. We are proposing a similar interaction between a user and computer, where sometimes the user leads and at other times the computer according to the context of the interaction. This contrasts with most current approaches where one agent, be it the human or the computer, pre-empts the other and where most interaction is driven by events and proceeds to varying degrees in rigid, over specified waysExample scenario: Exploring a Digitally Enriched EnvironmentAn example of how this approach could be used is that of location-aware information acquisition while walking in a town centre. You might feel a 'tick' on your phone's vibration motor, making you aware that there is information available about something in your environment. Your rich context understanding abilities would tell you how likely this 'tick' was to be of interest, if you ignore the cue and walk on, the negotiation would end there and then. If you are curious, you might gesture with the phone at likely targets in your surroundings, and get a response from several of them. If you are further intrigued, you may continue to interact with these potential targets, possibly moving from the vibro-tactile to an audio display, gaining information by an active exploration of the environment, something we have evolved to do naturally. The user explores the possibilities in the situation by directly engaging (probing or playing) with it, being able to move at will through the space of possibilities, gaining more and more insight during the interaction. The multimodal feedback provided both encodes the system's current interpretation of the user's intention (e.g. moving towards a target) and the probability of the target meeting the user's needs. After working through combinations of vibration and audio, if the joint dynamics of information source, and user continue to intertwine, the display of the mobile device might be used for full details. This example shows a 'schedule' of modalities, and illustrates the negotiation process in a practical and commercially interesting example.