HCC: Small: New Infrastructure Concepts for Robust Handling of Inputs with Uncertainty

目前用于处理几乎所有现代图形用户界面(GUI)中的输入的传统软件是有效且高度发展的。这具有促进重用而不是重新发明交互技术的优点，并使创建图形用户界面变得容易，即使对于编程能力有限的人也是如此。然而，这些成功的软件抽象假设报告给系统的输入准确地反映了用户的操作--输入是确定的，而不是不确定的。不幸的是，这并不适用于一些最有趣的新输入技术，包括自然输入，如自由空间手势(例如，通过Kinect深度摄像头检测到的)、笔输入(包括手写、手势和手绘)、触摸输入、上下文传感器和语音输入。其中一些新技术包含固有的不确定性，例如当手指触摸区域(用户看不到)比显示器的像素大得多时。其他人则使用识别器进行输入，通常会对可能发生的情况进行估计。由于传统的输入处理方法无法管理输入中的不确定性，其中许多方法甚至在输入处理开始之前就强制解决不确定性。例如，可以使用单个点(其质心)将来自触摸屏的输入的位置表示为确定的。但当不确定信息被丢弃时，界面可能很快就会变得脆弱；微小的识别错误可能会破坏交互并破坏用户体验。因此，事实证明，这些新的、非常有希望的投入形式往往很难充分发挥其潜力。PI在这个项目中的目标是通过创建重新设计的输入处理基础设施来克服这个问题，该基础设施将健壮地对具有不确定性的输入进行建模和利用。它将通过在概率的基础上处理所有输入以及源自该输入的所有UI操作，考虑对输入的多种可能的解释(以及随着时间的推移而产生的所有结果)，以及对每种解释的可能性的估计来实现这一点。因此，当需要作出决定和采取不可逆转的行动时，系统将有一个良好的基础来选择解释。PI的方法不是从全新的输入概念开始，而是通过支持不确定性来扩展传统的输入抽象。通常，单个特定的输入事件被分派给单个交互参与者，该交互参与者解释其含义以跟踪其自身的交互状态并最终请求操作。现在，输入过程的每个部分都将以概率方式完成。估计的概率分布将跟踪可能已经发生的输入备选方案、可能已经接收到该输入的交互角色、交互角色可能所处的状态以及交互角色可能因此请求的行动。然后，可以使用这些概率分布来做出关于何时、是否以及实际采取哪些行动的明智决策。为了对UI程序员隐藏随时间保持这些分布中的每一个的复杂性，PI将使用概率分布的蒙特卡罗表示(即，每个样本的加权集合指示一个确定值的概率)。至关重要的是，这种表示将允许代码简单地多次执行传统的(某些)输入处理步骤--对相关概率分布中的每个样本执行一次(S)。这几乎隐藏了与不确定性相关的所有复杂性，并允许程序员在输入处理的大多数方面使用他们当前的概念模型，甚至是与他们当前实践几乎相同的代码。广泛的影响：项目成果将从根本上改变将现成的新输入技术整合到交互系统中的容易程度，因此将对扩展我们构建和部署具有新输入形式的界面的能力产生广泛影响。作为这项研究的一部分，PI将为图形用户界面和上下文感知应用程序开发工作解决方案。他还将创建并广泛分发体现这些概念的完整教学工具包(其中，术语“教学”的使用精神与Pascal是一种教学编程语言的精神相同-它使用了良好的概念和当时的最佳实践；它在概念上是干净的，但适合实际工作)。这一教学工具包将纳入国际和平研究所的教育活动，并将开发课程单元，以便将其推广到其他大学。

The conventional software currently used to handle input in nearly all modern graphical user interfaces (GUIs) is effective and highly evolved. This has the advantages of promoting reuse rather than reinvention of interaction techniques, and making it easy to create GUis, even for those with limited programming ability. However, these successful software abstractions assume the inputs reported to the system accurately reflect the actions of the user - that input is certain rather than uncertain. Unfortunately, this does not hold for some of the most interesting new input technologies including naturalistic inputs such as free space gestures (e.g., as sensed by the Kinect depth camera), pen input (including handwriting, gestures, and free hand drawing), touch input, sensors for context, and voice input. Some of these new technologies contain inherent uncertainty, such as when a finger touch area (that the user cannot see) is much larger than the pixels of a display. Others make use of recognizers for input and typically produce estimates of what might have occurred. Since conventional methods of input handling have no way to manage uncertainty in input, many of them force uncertainty to be resolved before input processing even starts. For example, the location of input from a touch screen may be represented as certain using a single point (its centroid). But when uncertainty information is thrown away, interfaces can quickly become brittle; small recognition errors can derail the interaction and destroy the user experience. As a result, these new and very promising forms of input have often proven difficult to use to their full potential. The PI's goal in this project is to overcome this problem by creating a redesigned input-handling infrastructure, which will robustly model, and make use of, inputs with uncertainty. It will do this by treating all input, and all UI actions stemming from that input, on a probabilistic basis, entertaining multiple possible interpretations of input (and all its consequences over time), along with estimates of the likelihood of each interpretation. As a result, when decisions need to be made and irreversible actions undertaken, systems will have a sound basis for choosing among interpretations. Rather than starting with completely new input concepts, the PI's approach is to extend conventional input abstractions with support for uncertainty. Normally, a single certain input event is dispatched to a single interactor, which interprets its meaning to track its own interactive state and eventually request actions. Now, each of these parts of the input process will be done probabilistically. An estimated probability distribution will be tracked over input alternatives that might have occurred, interactors which might have received that input, states that interactors might be in, and actions that interactors might request as a result. These probability distributions can then be used to make informed decisions about when, whether, and which actions to actually undertake. To hide the complexity of maintaining each of these distributions over time from the UI programmer, the PI will employ a Monte Carlo representation of a probability distribution (i.e., a weighted set of samples each indicating the probability of one definite value). Crucially, this representation will allow the code to simply execute traditional (certain) input processing steps multiple times - once for each sample in the relevant probability distribution(s). This hides nearly all the complexity associated with uncertainty, and allows programmers to use their current conceptual models, and even code nearly identical to their current practices, for most aspects of input handling.Broader Impacts: Project outcomes will radically change the ease with which readily available new input technologies can be incorporated into interactive systems, and thus will have wide impact in expanding our ability to build and deploy interfaces with new forms of input. As part of this research, the PI will develop working solutions for both graphical user interfaces and context-aware applications. He will also create and widely distribute a full teaching toolkit which embodies these concepts (where the term "teaching" is used in the same spirit that Pascal was a teaching programming language - it used good concepts and the best practices of the time; it was conceptually clean, yet suitable for real work). This teaching toolkit will be integrated into educational activities at the PI's institution, and curricular modules will be developed which should allow this to be carried to other universities.