CAREER: Informal Tools for Multimodal User Interface Design

这是一个为期4年的连续奖励的第一年。多模态用户界面很可能是未来的主要界面。这些接口可以利用各种各样的人类感知和运动能力，如语音和手势，以实现更自然的人机通信。这个项目解决了目前在建立这种接口方面的困难。P1将创建新的交互式工具，用于设计针对各种设备的多模式用户界面，从掌上电脑到桌面，再到墙壁大小的显示器。这些工具将支持迭代设计方法，支持设计师目前在设计早期阶段使用的非正式技术，如草图，故事板和“绿野仙踪”。通过演示编程将允许指定不基于屏幕的事件，例如语音I/O或身体姿势输入。其目标是让设计师快速创建他们的想法的“粗略切割”，可以快速测试和修改。当准备就绪时，设计人员可以使用这些工具为各种设备生成用户界面代码。然后，可以使用最终的接口细节和应用程序代码来扩展此代码。作为一个示例应用程序，PI将应用新的工具来开发一套非正式的课堂演讲工具。他将进行非正式和受控实验，以评估设计工具的有效性，并对工具和示例应用程序进行迭代改进。这项研究将导致新的技术，用于指定视觉和非视觉输入，这将适用于多模态和单峰界面设计工具，使人们可以更容易地创建界面比目前的情况。研究结果还将影响残疾人（特别是视力受损者）以及在非传统环境（如移动的和可穿戴计算）中工作的主流用户的系统。

This is the first year of funding of a 4 year continuing award. Multimodal user interfaces are likely to be the predominant interface of the future. These interfaces can take advantage of a wide variety of human perceptual and motor abilities, such as speech and gesture, to achieve a more natural human computer communication. This project addresses the current difficulty in building such interfaces. The P1 will create new interactive tools for designing multimodal user interfaces targeted at a range of devices, from the palmtop to the desktop to wall sized displays. These tools will support an iterative design methodology which supports the informal techniques designers currently use in the early stages of design, such as sketching, storyboarding, and "Wizard of Oz". Programming by demonstration will allow specification of events that are not screen based, such as speech I/O or body gesture input. The goal is to allow designers to rapidly create "rough cuts" of their ideas, which can be quickly tested and modified. When ready, the designer can use these tools to generate user interface code for a variety of devices. This code can then be extended with the final interface details and application code. As an example application, the PI will apply the new tools to develop a set of informal presentation tools for classroom lectures. He will carry out both informal and controlled experiments to evaluate the effectiveness of the design tools and to inforin iterative refinement of both the tools and the example application. This research will result in new techniques for specifying visual and nonvisual input, which will be applicable to both multimodal and unimodal interface design tools, so that people can more easily create interfaces than is currently the case. The findings will also impact systems for people with disabilities (especially the vision impaired), as well as for mainstream users working in non traditional settings such as mobile and wearable computing.