Multimodal, Negotiated Interaction in Mobile Scenarios

我们的建议是研究一种替代方法，允许用户在他们的环境中与内容和服务进行交互，这样他们所做的动作、动作、手势等，以及他们收到的反馈都是连续的，用户和系统以一种流畅、动态的方式协商他们的交互。我们认为恰当的比喻是跳舞，而不是当前的命令与控制比喻。当一个人与舞伴跳舞时，会有一种柔和的控制起伏；有时一个人领舞，有时另一个人领舞，这种变化随着他们跳舞而流畅地变化。我们建议在用户和计算机之间进行类似的交互，有时用户引导，有时计算机根据交互的上下文引导。这与大多数当前的方法形成对比，其中一个代理，无论是人还是计算机，先发制人，大多数交互是由事件驱动的，并以不同程度的刚性，超过指定的方式进行。示例场景：探索数字丰富的环境。如何使用这种方法的一个例子是在城镇中心行走时获取位置感知信息。你可能会感觉到手机的振动马达发出“滴答声”，让你意识到周围环境中有一些信息可用。你丰富的上下文理解能力会告诉你这个“勾”有多大可能是你感兴趣的，如果你忽略了这个提示，继续走下去，谈判就会在那里结束。如果你很好奇，你可以用手机对着周围可能的目标做手势，从中得到几个人的回应。如果你进一步感兴趣，你可能会继续与这些潜在的目标互动，可能会从振动触觉转向音频显示，通过对环境的积极探索获得信息，这是我们进化的自然行为。用户通过直接参与（探索或玩游戏）来探索情境中的可能性，能够在可能性空间中随意移动，在互动过程中获得越来越多的洞察力。提供的多模态反馈既编码了系统当前对用户意图的解释（例如向目标移动），也编码了目标满足用户需求的可能性。在完成振动和音频的组合后，如果信息源和用户的联合动态继续纠缠，那么移动设备的显示可能会被用于完整的细节。这个例子展示了模式的“时间表”，并以一个实际的和商业上有趣的例子说明了谈判过程。

Our proposal is to investigate an alternative means of allowing users to interact with content and services in their environment such that the actions they make, movements, gestures, etc., and feedback they receive are continuous, with the user and system negotiating their interactions in a fluid, dynamic way. We believe the appropriate comparison would be dancing, rather than the current command & control metaphor. When someone dances with a partner there is a soft ebb and flow of control; sometimes one person leads, sometimes the other, this changing fluidly as they dance. We are proposing a similar interaction between a user and computer, where sometimes the user leads and at other times the computer according to the context of the interaction. This contrasts with most current approaches where one agent, be it the human or the computer, pre-empts the other and where most interaction is driven by events and proceeds to varying degrees in rigid, over specified waysExample scenario: Exploring a Digitally Enriched EnvironmentAn example of how this approach could be used is that of location-aware information acquisition while walking in a town centre. You might feel a 'tick' on your phone's vibration motor, making you aware that there is information available about something in your environment. Your rich context understanding abilities would tell you how likely this 'tick' was to be of interest, if you ignore the cue and walk on, the negotiation would end there and then. If you are curious, you might gesture with the phone at likely targets in your surroundings, and get a response from several of them. If you are further intrigued, you may continue to interact with these potential targets, possibly moving from the vibro-tactile to an audio display, gaining information by an active exploration of the environment, something we have evolved to do naturally. The user explores the possibilities in the situation by directly engaging (probing or playing) with it, being able to move at will through the space of possibilities, gaining more and more insight during the interaction. The multimodal feedback provided both encodes the system's current interpretation of the user's intention (e.g. moving towards a target) and the probability of the target meeting the user's needs. After working through combinations of vibration and audio, if the joint dynamics of information source, and user continue to intertwine, the display of the mobile device might be used for full details. This example shows a 'schedule' of modalities, and illustrates the negotiation process in a practical and commercially interesting example.