Reality-Based Interaction: A New Framework for Understanding the Next Generation of Human-Computer Interfaces

当今人机交互的实践状态是图形用户界面，或直接操作，交互风格。该项目旨在从一系列看似脱节的当前研究（包括虚拟现实，增强现实，无处不在，普及和手持交互，有形用户界面，轻量级，隐性或被动交互，感知界面，情感计算，上下文感知界面，语音和多模态界面）中识别和统一新兴的下一代用户交互。无处不在的计算、有形的界面以及计算机在各种产品和对象中的普及，正在改变与计算机的交互，使其从一种与日常生活分离的专门活动，越来越成为现实世界的一部分。与此同时，随着计算机越来越成为现实世界的一部分，用户界面似乎越来越像现实世界（例如，在虚拟现实中）。PI相信这些可以通过他的“基于现实的交互”的概念联系起来，通过关注基于现实的界面如何利用用户的内在能力，并且这个想法将提供必要的杠杆将它们联系在一起并定义新一代的用户界面，希望这个天真的概念可以构建成一个有用的理论框架。从自然的或“基于现实的”界面的概念出发，因为它们利用了用户已经拥有的能力而获得了力量，PI将形式化学习知识与知识的概念。使用系统所需的“基于现实的”技能。他将把它发展成一个理论框架，并更正式地充实它，并用它来确定具体的开放问题进行调查。然后，PI将根据需要修改或重新发明初始方法，并将测试理论，首先通过将其应用于一系列已发表的结果，然后通过设计专门用于测试的实验，其中用户界面的各个方面可以选择性地操纵实验。在这一阶段的工作中，项目负责人将设计、实施和评估框架提出的代表差距或机会的新交互技术。更广泛的影响：本项目将为下一代人机界面提供一个统一的理论框架，作为该领域研究的基础和指导。它还将导致新交互技术的原型实现和实验评估，并将进一步帮助扩展新交互风格的研究，而不是目前可以追求它的相对较大和资金充足的机构，因为PI将开发一个廉价的工作台（或基础设施包）作为项目的一部分来支持这项工作，然后他将在网络上传播（硬件描述和软件工具）给其他大学。

The state of practice in human-computer interaction today is the graphical user interface, or direct manipulation, interaction style. This project seeks to identify and unify an emerging next generation of user interaction from a range of seemingly disconnected current research (including virtual reality, augmented reality, ubiquitous, pervasive, and handheld interaction, tangible user interfaces, lightweight, tacit, or passive interaction, perceptual interfaces, affective computing, context-aware interfaces, and speech and multi-modal interfaces). Ubiquitous computing, tangible interfaces, and the spread of computers into a wide range of products and objects are changing interacting with computers, from a specialized activity segregated from daily life, to one that is more and more a part of the real world. At the same time, as computers are becoming more a part of the real world, user interfaces seem to be evolving to behave more and more like the real world (e.g., in virtual reality). The PI believes these can be connected through his concept of "reality-based interaction" by focusing on the ways in which interfaces that are based on reality exploit users' built-in abilities, and that this idea will provide the necessary leverage to tie them together and define a new generation of user interfaces, in the hope that this naive notion can be built into a useful theoretical framework. Starting from the idea of natural or "reality-based" interfaces, which gain their strength because they exploit abilities that their users already possess, the PI will formalize the notion of learned knowledge vs. "reality-based" skills needed to use a system. He will develop this into a theoretical framework and flesh it out more formally, and use it to identify specific open issues for investigation. The PI will then modify or reinvent the initial approach, as needed, and will test the theory, first by applying it against a range of published results, and second by devising experiments specifically for testing, where aspects of a user interface can be selectively manipulated for the experiment. For this phase of the work, the PI will design, implement, and evaluate selected new interaction techniques representing gaps or opportunities suggested by the framework.Broader Impacts: This project will lead to a unified theoretical framework for the next generation of human-computer interfaces, which should serve as a foundation and guide for research in the field. It will also result in prototype implementations and experimental evaluation of new interaction techniques, and will furthermore help extend research in new interaction styles beyond the relatively large and well-funded institutions that can currently pursue it, in that the PI will develop as part of the project an inexpensive workbench (or infrastructure package) to support this work, which he will then disseminate on the web (hardware descriptions and software tools) to other universities.