CAREER: Informal Tools for Multimodal User Interface Design

这是四年持续奖的第一年资助。多模式用户界面很可能是未来的主流界面。这些接口可以利用人类的各种感知和运动能力，如语音和手势，实现更自然的人机交流。该项目解决了目前在构建此类接口方面的困难。P1将创建新的交互工具，用于设计针对一系列设备的多模式用户界面，从掌上电脑到桌面再到墙壁大小的显示器。这些工具将支持迭代设计方法，支持设计者目前在设计早期阶段使用的非正式技术，如草图、故事板和《绿野仙踪》。通过演示编程将允许指定不基于屏幕的事件，例如语音I/O或身体手势输入。这样做的目的是让设计师能够快速创建他们的想法的“粗剪”，可以快速地进行测试和修改。准备好后，设计人员可以使用这些工具为各种设备生成用户界面代码。然后可以使用最终的接口细节和应用程序代码来扩展该代码。作为一个示例应用程序，PI将应用新工具来开发一套用于课堂讲课的非正式演示工具。他将进行非正式和受控实验，以评估设计工具的有效性，并为工具和示例应用程序的迭代改进提供信息。这项研究将产生指定视觉和非视觉输入的新技术，这种技术将适用于多通道和单通道界面设计工具，使人们能够比目前更容易地创建界面。这些发现还将影响残疾人(特别是视力障碍者)以及在移动和可穿戴计算等非传统环境中工作的主流用户的系统。

This is the first year of funding of a 4 year continuing award. Multimodal user interfaces are likely to be the predominant interface of the future. These interfaces can take advantage of a wide variety of human perceptual and motor abilities, such as speech and gesture, to achieve a more natural human computer communication. This project addresses the current difficulty in building such interfaces. The P1 will create new interactive tools for designing multimodal user interfaces targeted at a range of devices, from the palmtop to the desktop to wall sized displays. These tools will support an iterative design methodology which supports the informal techniques designers currently use in the early stages of design, such as sketching, storyboarding, and "Wizard of Oz". Programming by demonstration will allow specification of events that are not screen based, such as speech I/O or body gesture input. The goal is to allow designers to rapidly create "rough cuts" of their ideas, which can be quickly tested and modified. When ready, the designer can use these tools to generate user interface code for a variety of devices. This code can then be extended with the final interface details and application code. As an example application, the PI will apply the new tools to develop a set of informal presentation tools for classroom lectures. He will carry out both informal and controlled experiments to evaluate the effectiveness of the design tools and to inforin iterative refinement of both the tools and the example application. This research will result in new techniques for specifying visual and nonvisual input, which will be applicable to both multimodal and unimodal interface design tools, so that people can more easily create interfaces than is currently the case. The findings will also impact systems for people with disabilities (especially the vision impaired), as well as for mainstream users working in non traditional settings such as mobile and wearable computing.