Multimodal, Negotiated Interaction in Mobile Scenarios

我们的建议是研究一种替代方法，允许用户与其环境中的内容和服务进行交互，以便他们所做的动作，动作，手势等，并且它们接收的反馈是连续的，用户和系统以流畅、动态的方式协商它们的交互。我们认为，适当的比喻应该是跳舞，而不是目前的命令和控制比喻。当一个人和一个舞伴跳舞时，会有一种柔和的控制力的起伏;有时一个人带头，有时另一个人带头，这种控制力随着他们的舞蹈而流畅地变化。我们建议在用户和计算机之间进行类似的交互，有时用户引导，有时计算机根据交互的上下文引导。这与大多数当前的方法形成对比，其中一个代理，无论是人类还是计算机，都优先于另一个，并且大多数交互都由事件驱动，并且以严格的、指定的方式进行不同程度的处理示例场景：探索数字丰富的环境这种方法的一个示例是在市中心行走时的位置感知信息获取。你可能会感觉到手机振动马达上的“滴答声”，让你意识到你周围的环境中有一些信息。你丰富的上下文理解能力会告诉你这个“滴答”是多么有趣，如果你无视这个提示，继续走下去，谈判就会就此结束。如果你很好奇，你可以用手机对周围可能的目标做手势，然后得到其中几个目标的回应。如果你进一步感兴趣，你可以继续与这些潜在的目标互动，可能从振动触觉移动到音频显示，通过对环境的积极探索来获取信息，这是我们自然进化的结果。用户通过直接参与（探索或玩耍）来探索情境中的可能性，能够在可能性的空间中随意移动，在交互过程中获得越来越多的洞察力。所提供的多模态反馈对系统对用户意图（例如，朝向目标移动）的当前解释和目标满足用户需求的概率进行编码。在完成振动和音频的组合后，如果信息源和用户的联合动态继续交织，则可以使用移动终端的显示来获取完整的细节。这个例子显示了一个“时间表”的方式，并说明了谈判过程中的一个实际和商业上有趣的例子。

Our proposal is to investigate an alternative means of allowing users to interact with content and services in their environment such that the actions they make, movements, gestures, etc., and feedback they receive are continuous, with the user and system negotiating their interactions in a fluid, dynamic way. We believe the appropriate comparison would be dancing, rather than the current command & control metaphor. When someone dances with a partner there is a soft ebb and flow of control; sometimes one person leads, sometimes the other, this changing fluidly as they dance. We are proposing a similar interaction between a user and computer, where sometimes the user leads and at other times the computer according to the context of the interaction. This contrasts with most current approaches where one agent, be it the human or the computer, pre-empts the other and where most interaction is driven by events and proceeds to varying degrees in rigid, over specified waysExample scenario: Exploring a Digitally Enriched EnvironmentAn example of how this approach could be used is that of location-aware information acquisition while walking in a town centre. You might feel a 'tick' on your phone's vibration motor, making you aware that there is information available about something in your environment. Your rich context understanding abilities would tell you how likely this 'tick' was to be of interest, if you ignore the cue and walk on, the negotiation would end there and then. If you are curious, you might gesture with the phone at likely targets in your surroundings, and get a response from several of them. If you are further intrigued, you may continue to interact with these potential targets, possibly moving from the vibro-tactile to an audio display, gaining information by an active exploration of the environment, something we have evolved to do naturally. The user explores the possibilities in the situation by directly engaging (probing or playing) with it, being able to move at will through the space of possibilities, gaining more and more insight during the interaction. The multimodal feedback provided both encodes the system's current interpretation of the user's intention (e.g. moving towards a target) and the probability of the target meeting the user's needs. After working through combinations of vibration and audio, if the joint dynamics of information source, and user continue to intertwine, the display of the mobile device might be used for full details. This example shows a 'schedule' of modalities, and illustrates the negotiation process in a practical and commercially interesting example.