HCC: Small: Enabling and Exploring Natural Interaction

技术的进步使得在各种设备和环境中探索人机交互的新方法成为可能，以实现更自然的交互。 虽然从用户和商业角度来看都很有价值，但从更有原则的系统设计或认知科学角度来看，新技术的使用并没有得到很好的理解。 即使建立一套设计原则的基本要素，也可以产生更好的设计，并增强我们在高风险/高回报领域（例如急救人员规划和控制）使用该技术的信心。 PI 的长期目标是制定一套原则，详细说明如何设计既能实现自然人机交互又能了解人为因素的系统。 自然交互是指人与人之间可以发生的认知透明、轻松的多模式沟通；这项工作旨在使人机交互成为可能。 以人为因素为依据的设计考虑到对人类能力的理解（例如，注意力、多个信息通道的使用等），以便最终系统与人类信息处理具有良好的阻抗匹配。 这项研究将涉及构建本着这种精神设计的系统并阐明其设计原则，这些原则反过来将通过明确一种或另一种模式合适的任务条件（例如何时画画、何时说话）以及在人机通信中有效同时使用多种模式的方式来促进未来的设计。 该项目以桌面系统为背景，协助城市搜救 (USAR) 行动指挥中心的规划和协调。 这项工作将通过利用和结合团队在构建新型交互技术和人为因素方面的经验来进行。 他们将扩展 PI 桌面系统的当前版本，该系统允许基本的笔式交互，从而使其能够处理现实世界 USAR 工作中使用的草图、徒手手势和语音，从而提供更自然的交互方式。 额外的交互方式将使系统更加强大，而任务的现实世界、时间压力特征为研究人为因素方面提供了一个良好的平台。 这将使团队能够了解各种模式如何、何时以及为何有用，并提供可以阐明系统设计原则的数据。 为了帮助确保项目成果的广泛适用性，PI 将在第二个领域探索相同的问题，即使用 UML 图进行软件设计。 智力优势：这项研究将通过产生有关模态选择的经验数据，并通过阐明一套广泛有用的界面设计原则来深入了解多模态交互模型，这些原则明确了模态选择和认知有效模态组合的条件。 这些原则为多模式界面设计提供了变革性改变的可能性，将其从当前很大程度上临时的实践转变为以可测试原则为指导的设计过程。 在两个任务领域进行研究将有助于确保所得出的原理具有有用的通用性。 这项工作将为社会带来好处，因为它可以提高急救人员团队的效率。 处理非传统交互方式（例如手势）的能力最终将使身体状况不佳的用户更容易进行计算机交互。 PI 将尽可能注意使用和构建开源工具，以便他可以向社区提供项目过程中产生的所有研究软件，从而增加下一代研究和教育平台的供应。

Advances in technology are making it feasible to explore novel approaches to human-computer interaction in a wide variety of devices and settings, in an effort to achieve interaction that feels more natural. While valuable from both a user and commercial perspective, use of new technology is not well understood from a more principled system design or cognitive science perspective. Establishing even the basic elements of a set of design principles would both produce better designs and increase our confidence in using the technology in high-risk/high-reward domains, such as first responder planning and control. The PI's long-term goal is to develop a set of principles specifying how to design systems that both enable natural human-computer interaction and are informed by an understanding of human factors. Natural interaction refers to the cognitively transparent, effortless multimodal communication that can happen between people; this work aims to make that possible in human-computer interaction. Designs informed by human factors take into account an understanding of human capabilities (e.g., attention, use of multiple information channels, etc.), so that the final system is a good impedance match to human information processing. This research will involve building systems designed in this spirit and articulating principles for their design that, in turn, will facilitate future designs by making explicit both the task conditions under which one or another modality is appropriate (e.g., when to draw, when to talk), and the ways in which multiple modalities can effectively be used simultaneously in human-computer communication. The project is set in the context of a tabletop-based system that assists with planning and coordination in the command center of an urban search and rescue (USAR) operation. The work will proceed by leveraging and combining the team's experience in building novel interaction technologies and in human factors. They will extend the current version of the PI's tabletop system, which permits basic pen-based interaction, so as to give it the ability to handle the kinds of sketching, freehand gestures and speech used in real-world USAR work, thereby providing a far more natural style of interaction. The additional interaction modalities will make the system more powerful, while the real-world, time-pressured character of the task offers a good platform for studying the human factors aspects. This will allow the team to understand how, when and why various modalities are useful, providing the data from which system design principles can be articulated. To help ensure breadth of applicability of project outcomes, the PI will explore the same issues in a second domain, software design with UML diagrams.Intellectual Merit: This research will provide insight into a model of multimodal interaction by producing empirical data about modality selection, and by articulating a widely useful set of principles for interface design that make explicit the conditions for both modality selection and cognitively effective modality combination. Such principles offer the possibility of transformative change to multimodal interface design, changing it from the current largely ad hoc practice to a design process guided by testable principles. Pursuing the research in two task domains will help to ensure a useful degree of generality to the principles derived. The work will provide benefit to society to the extent it can improve the effectiveness of first responder teams. The ability to handle non-traditional interaction modalities (e.g., gesture) will ultimately make computer interaction more accessible to physically disadvantaged users. The PI will take care, to the extent possible, to use and build upon open source tools, so that he can make available to the community all of the research software produced during the course of the project, thereby adding to the supply of next-generation research and education platforms.