HCC: Small: Enabling and Exploring Natural Interaction

技术的进步使得在各种设备和设置中探索人机交互的新方法变得可行，以实现感觉更自然的交互。 虽然从用户和商业的角度来看都是有价值的，但从更有原则的系统设计或认知科学的角度来看，新技术的使用并没有得到很好的理解。 即使是建立一套设计原则的基本要素，也会产生更好的设计，并增加我们在高风险/高回报领域使用该技术的信心，例如第一响应者规划和控制。 PI的长期目标是制定一套原则，规定如何设计既能实现自然人机交互又能了解人为因素的系统。 自然交互是指人与人之间可以进行的认知透明，毫不费力的多模式通信;这项工作旨在使人机交互成为可能。 以人为因素为依据的设计考虑到对人的能力的理解（例如，注意、使用多个信息渠道等），使得最终的系统与人类信息处理具有良好的阻抗匹配。 这项研究将涉及以这种精神设计的建筑系统，并阐明其设计原则，反过来，通过明确一种或另一种模式适合的任务条件（例如，何时绘画、何时说话），以及在人机通信中可以同时有效地使用多种模态的方式。 该项目是在一个基于桌面的系统的背景下，协助规划和协调在指挥中心的城市搜索和救援（USAR）行动。 这项工作将通过利用和结合团队在构建新型交互技术和人为因素方面的经验来进行。 他们将扩展PI桌面系统的当前版本，该系统允许基本的基于笔的交互，以便使其能够处理真实世界USAR工作中使用的各种草图，徒手手势和语音，从而提供更自然的交互风格。 额外的交互方式将使系统更强大，而现实世界中，时间紧迫的任务的字符提供了一个很好的平台，研究人的因素方面。 这将使团队能够了解各种模式如何、何时以及为什么有用，并提供可以阐明系统设计原则的数据。 为了帮助确保项目成果的广泛适用性，PI将在第二个领域探索相同的问题，即使用UML图进行软件设计。本研究将通过产生关于情态选择的经验数据，并阐明了一套广泛适用的界面设计原则，明确了模态选择和认知有效性的条件模态组合 这些原则为多模态界面设计提供了变革的可能性，将其从目前主要是临时的实践转变为可测试原则指导的设计过程。 在两个任务领域进行研究将有助于确保所得出的原则具有有用的普遍性。 这项工作将为社会带来好处，因为它可以提高第一反应小组的效率。 处理非传统交互模式的能力（例如，手势）将最终使身体上处于不利地位的用户更容易获得计算机交互。 PI将尽可能地使用和构建开源工具，以便他可以向社区提供项目过程中产生的所有研究软件，从而增加下一代研究和教育平台的供应。

Advances in technology are making it feasible to explore novel approaches to human-computer interaction in a wide variety of devices and settings, in an effort to achieve interaction that feels more natural. While valuable from both a user and commercial perspective, use of new technology is not well understood from a more principled system design or cognitive science perspective. Establishing even the basic elements of a set of design principles would both produce better designs and increase our confidence in using the technology in high-risk/high-reward domains, such as first responder planning and control. The PI's long-term goal is to develop a set of principles specifying how to design systems that both enable natural human-computer interaction and are informed by an understanding of human factors. Natural interaction refers to the cognitively transparent, effortless multimodal communication that can happen between people; this work aims to make that possible in human-computer interaction. Designs informed by human factors take into account an understanding of human capabilities (e.g., attention, use of multiple information channels, etc.), so that the final system is a good impedance match to human information processing. This research will involve building systems designed in this spirit and articulating principles for their design that, in turn, will facilitate future designs by making explicit both the task conditions under which one or another modality is appropriate (e.g., when to draw, when to talk), and the ways in which multiple modalities can effectively be used simultaneously in human-computer communication. The project is set in the context of a tabletop-based system that assists with planning and coordination in the command center of an urban search and rescue (USAR) operation. The work will proceed by leveraging and combining the team's experience in building novel interaction technologies and in human factors. They will extend the current version of the PI's tabletop system, which permits basic pen-based interaction, so as to give it the ability to handle the kinds of sketching, freehand gestures and speech used in real-world USAR work, thereby providing a far more natural style of interaction. The additional interaction modalities will make the system more powerful, while the real-world, time-pressured character of the task offers a good platform for studying the human factors aspects. This will allow the team to understand how, when and why various modalities are useful, providing the data from which system design principles can be articulated. To help ensure breadth of applicability of project outcomes, the PI will explore the same issues in a second domain, software design with UML diagrams.Intellectual Merit: This research will provide insight into a model of multimodal interaction by producing empirical data about modality selection, and by articulating a widely useful set of principles for interface design that make explicit the conditions for both modality selection and cognitively effective modality combination. Such principles offer the possibility of transformative change to multimodal interface design, changing it from the current largely ad hoc practice to a design process guided by testable principles. Pursuing the research in two task domains will help to ensure a useful degree of generality to the principles derived. The work will provide benefit to society to the extent it can improve the effectiveness of first responder teams. The ability to handle non-traditional interaction modalities (e.g., gesture) will ultimately make computer interaction more accessible to physically disadvantaged users. The PI will take care, to the extent possible, to use and build upon open source tools, so that he can make available to the community all of the research software produced during the course of the project, thereby adding to the supply of next-generation research and education platforms.