GAIME: Gestural and Audio Interactions for Mobile Environments

大多数pda和智能手机都有复杂的图形界面，通常使用小键盘或手写笔进行输入。此类设备的应用和服务范围一直在增长。然而，当用户在移动时，存在一些问题会使交互变得困难。操作许多应用程序需要大量的视觉注意力，这在移动环境中可能不可用。Oulasvirta等人的研究表明，对于移动中的用户来说，注意力可能变得非常分散，因为它必须在导航环境和设备之间转换，这使得交互变得困难。我们自己的研究表明，当用户使用移动设备时，性能可能会下降20%以上。另一个重要的问题是，大多数设备需要手来操作许多应用程序。例如，当用户提着包、抱着孩子或操作机器时，这些可能无法使用。这项提议的新颖之处在于，通过研究身体其他部位的手势输入与三维声音输出相结合，减少了对图形显示和手的依赖。为移动用户提供免提输入和控制方面的工作很少。语音识别在这种情况下仍然存在问题，因为它的高处理要求和设备使用的动态音频环境。许多关于手势输入的研究仍然使用手来做出手势。在头部输入方面有一些工作，通常是为残疾用户做的，但在移动环境中很少使用。我们自己之前的工作已经开始研究头部指向，并表明它可能是一种有用的方式来指向和选择移动[3]。许多其他身体位置可能对移动时的微妙和谨慎输入有用（例如，用户行走或坐在颠簸的火车上）。例如，手腕旋转具有控制径向菜单的潜力，因为手腕可以旋转以在菜单上移动指针。它不引人注目，可以使用与手势相同的传感器进行跟踪（例如在手表中）。步态的微小变化也是相互作用的一种可能性。在之前的工作b[12]中，我们从PDA上的加速度计提取步态信息以查看可用性错误。我们可以调整这种技术，这样用户就可以稍微改变一个步骤的时间来进行输入。目前还没有对全身不同输入可能性的系统研究。我们将开发一种新的测试方法，使用菲茨定律分析以及更主观的措施来找出哪些身体位置对移动输入最有用。当用户移动时，由于视觉注意力的负荷，输出也是一个问题。我们和其他人已经开始研究在移动设备上使用空间化音频线索作为图像的替代或补充[1,6][19,32]。其中许多游戏使用非常简单的3D音频显示，但是，通过精心设计，空间音频可以提供更丰富的显示空间。我们的audio - clouds项目为3D音频交互建立了一些基础，研究了基本的指向行为、目标大小和分离[1,3]。我们现在需要推进这项工作，发展更复杂的相互作用。这里的关键方面是开发以自我为中心（固定于用户）和以外中心（固定于世界）显示的使用，以及如何将它们结合起来创建一个丰富的3D显示空间进行交互。这个项目的最后一个关键部分是创建引人注目的应用程序，将最好的音频和手势结合起来。然后，我们可以在更现实的环境中对用户进行更长时间的测试，以微调这些交互在现实世界中的工作方式。

Most PDAs and smart phones have sophisticated graphical interfaces and commonly use small keyboards or styli for input. The range of applications and services for such devices is growing all the time. However, there are problems which make interaction difficult when a user is on the move. Much visual attention is needed to operate many of the applications, which may not be available in mobile contexts. Oulasvirta et al. [29] showed that attention can become very fragmented for users on the move as it must shift between navigating the environment and the device, making interaction hard. Our own research has shown that performance may drop by more than 20% when users are mobile [4]. Another important issue is that most devices require hands to operate many of the applications. These may not be available if the user is carrying bags, holding on to children or operating machinery, for example. The novel aspect of this proposal is to reduce the reliance on graphical displays and hands by investigating gesture input from other locations on the body com-bined with three-dimensional sound for output.Little work has gone into making input and control hands-free for mobile users. Speech recognition is still problematic in such settings due to its high processing requirements and the dynamic audio environments in which devices are used. Much of the research on ges-ture input still uses hands for making the gestures. There is some work on head-based input, often for users with disabilities [26], but little of this has been used in mobile settings. Our own previous work has begun to examine head pointing and showed that it might be a useful way to point and select on the move [3].Many other body locations could be useful for subtle and discreet input whilst mobile (e.g., users walking or sitting on a bumpy train). For example, wrist rotation has potential for controlling a radial menu as the wrist can be rotated to move a pointer across the menu. It is unobtrusive and could be tracked using the same sensor used for hand pointing gestures (in a watch for exam-ple). Small changes in gait are also a possibility for in-teraction. In previous work [12] we extracted gait in-formation from an accelerometer on a PDA to look at usability errors. We can adapt this technique so that users could slightly change the timing of a step to make input. There has been no systematic study of the differ-ent input possibilities across the body. We will develop a novel testing methodology using a Fitts' law analysis along with more subjective measures to find out which body locations are most useful for input on the move.Output is also a problem due to the load on visual atten-tion when users are mobile. We and others have begun to look at the use of spatialised audio cues for output when mobile as an alternative or complement to graph-ics [1, 6] [19, 32]. Many of these use very simple 3D audio displays, but, with careful design, spatial audio could provide a much richer display space. Our Audio-Clouds project built some foundations for 3D audio interactions, investigating basic pointing behaviour, target size and separation [1,3]. We need to now take this work forward and develop more sophisticated inter-actions. Key aspects here are to develop the use of ego-centric (fixed to the user) and exocentric (fixed to the world) displays, and how they can be combined to cre-ate a rich 3D display space for interaction.The final key part of this project is to create compelling applications which combine the best of the audio and gestures. We can then test these with users in more realistic settings over longer time periods to fine-tune how these interactions work in the real world.