HCC: Small: New Infrastructure Concepts for Robust Handling of Inputs with Uncertainty

HCC：小型：用于稳健处理不确定性输入的新基础设施概念

• 批准号: 1217929
• 负责人: Scott Hudson
• 金额: $ 49.99万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2012
• 资助国家: 美国
• 起止时间: 2012-08-15 至 2016-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1217929&HistoricalAwards=false
• 关键词: HCC; Small; New; Infrastructure; Concepts

The conventional software currently used to handle input in nearly all modern graphical user interfaces (GUIs) is effective and highly evolved. This has the advantages of promoting reuse rather than reinvention of interaction techniques, and making it easy to create GUis, even for those with limited programming ability. However, these successful software abstractions assume the inputs reported to the system accurately reflect the actions of the user - that input is certain rather than uncertain. Unfortunately, this does not hold for some of the most interesting new input technologies including naturalistic inputs such as free space gestures (e.g., as sensed by the Kinect depth camera), pen input (including handwriting, gestures, and free hand drawing), touch input, sensors for context, and voice input. Some of these new technologies contain inherent uncertainty, such as when a finger touch area (that the user cannot see) is much larger than the pixels of a display. Others make use of recognizers for input and typically produce estimates of what might have occurred. Since conventional methods of input handling have no way to manage uncertainty in input, many of them force uncertainty to be resolved before input processing even starts. For example, the location of input from a touch screen may be represented as certain using a single point (its centroid). But when uncertainty information is thrown away, interfaces can quickly become brittle; small recognition errors can derail the interaction and destroy the user experience. As a result, these new and very promising forms of input have often proven difficult to use to their full potential. The PI's goal in this project is to overcome this problem by creating a redesigned input-handling infrastructure, which will robustly model, and make use of, inputs with uncertainty. It will do this by treating all input, and all UI actions stemming from that input, on a probabilistic basis, entertaining multiple possible interpretations of input (and all its consequences over time), along with estimates of the likelihood of each interpretation. As a result, when decisions need to be made and irreversible actions undertaken, systems will have a sound basis for choosing among interpretations. Rather than starting with completely new input concepts, the PI's approach is to extend conventional input abstractions with support for uncertainty. Normally, a single certain input event is dispatched to a single interactor, which interprets its meaning to track its own interactive state and eventually request actions. Now, each of these parts of the input process will be done probabilistically. An estimated probability distribution will be tracked over input alternatives that might have occurred, interactors which might have received that input, states that interactors might be in, and actions that interactors might request as a result. These probability distributions can then be used to make informed decisions about when, whether, and which actions to actually undertake. To hide the complexity of maintaining each of these distributions over time from the UI programmer, the PI will employ a Monte Carlo representation of a probability distribution (i.e., a weighted set of samples each indicating the probability of one definite value). Crucially, this representation will allow the code to simply execute traditional (certain) input processing steps multiple times - once for each sample in the relevant probability distribution(s). This hides nearly all the complexity associated with uncertainty, and allows programmers to use their current conceptual models, and even code nearly identical to their current practices, for most aspects of input handling.Broader Impacts: Project outcomes will radically change the ease with which readily available new input technologies can be incorporated into interactive systems, and thus will have wide impact in expanding our ability to build and deploy interfaces with new forms of input. As part of this research, the PI will develop working solutions for both graphical user interfaces and context-aware applications. He will also create and widely distribute a full teaching toolkit which embodies these concepts (where the term "teaching" is used in the same spirit that Pascal was a teaching programming language - it used good concepts and the best practices of the time; it was conceptually clean, yet suitable for real work). This teaching toolkit will be integrated into educational activities at the PI's institution, and curricular modules will be developed which should allow this to be carried to other universities.

目前用于处理几乎所有现代图形用户界面（GUI）中的输入的传统软件是有效的并且高度进化的。 这样做的好处是促进了交互技术的重用，而不是重新发明，并且使得创建GUI变得容易，即使对于那些编程能力有限的人来说也是如此。 然而，这些成功的软件抽象假设报告给系统的输入准确地反映了用户的行为-输入是确定的而不是不确定的。 不幸的是，这并不适用于一些最有趣的新输入技术，包括自然输入，诸如自由空间手势（例如，如由Kinect深度相机感测到的）、笔输入（包括手写、手势和徒手画）、触摸输入、用于上下文的传感器和语音输入。 这些新技术中的一些包含固有的不确定性，例如当手指触摸区域（用户无法看到）比显示器的像素大得多时。 其他人使用识别器作为输入，通常对可能发生的事情进行估计。 由于传统的输入处理方法无法管理输入中的不确定性，因此许多输入处理方法甚至在输入处理开始之前就强制解决不确定性。 例如，来自触摸屏的输入的位置可以使用单个点（其质心）来表示为确定的。 但是，当不确定性信息被丢弃时，界面可能很快变得脆弱;小的识别错误可能会破坏交互并破坏用户体验。 因此，这些新的和非常有前途的投入形式往往证明难以充分发挥其潜力。 PI在这个项目中的目标是通过创建一个重新设计的输入处理基础设施来克服这个问题，该基础设施将鲁棒地建模并利用具有不确定性的输入。 它将通过在概率的基础上处理所有输入以及源于该输入的所有UI操作来实现这一点，考虑输入的多种可能解释（以及随着时间的推移其所有后果），以及对每种解释的可能性的估计沿着。 因此，当需要作出决定和采取不可逆转的行动时，系统将有一个在各种解释中作出选择的良好基础。 PI的方法不是从全新的输入概念开始，而是通过支持不确定性来扩展传统的输入抽象。 通常情况下，单个特定的输入事件被分派给单个交互器，交互器解释其含义以跟踪其自己的交互状态并最终请求操作。 现在，输入过程的每一部分都将以概率的方式完成。 估计的概率分布将在可能已经发生的输入替代、可能已经接收到该输入的交互者、交互者可能处于的状态以及交互者作为结果可能请求的动作上被跟踪。 然后，这些概率分布可以用于做出关于何时、是否以及实际采取哪些行动的明智决策。 为了向UI程序员隐藏随时间维护这些分布中的每一个的复杂性，PI将采用概率分布的蒙特卡罗表示（即，每个指示一个确定值的概率的样本的加权集合）。 至关重要的是，这种表示将允许代码简单地多次执行传统的（某些）输入处理步骤-对于相关概率分布中的每个样本执行一次。 这几乎隐藏了与不确定性相关的所有复杂性，并允许程序员使用他们当前的概念模型，甚至在输入处理的大多数方面使用与他们当前实践几乎相同的代码。 项目成果将从根本上改变现有的新输入技术可以被纳入交互系统的容易程度，因此将对扩展我们构建和部署具有新输入形式的界面的能力产生广泛的影响。 作为这项研究的一部分，PI将为图形用户界面和上下文感知应用程序开发工作解决方案。 他还将创建并广泛分发一个包含这些概念的完整教学工具包（其中“教学”一词的使用与Pascal作为教学编程语言的精神相同-它使用了当时的良好概念和最佳实践;它在概念上是干净的，但适合真实的工作）。 该教学工具包将被纳入公共教育机构的教育活动，并将开发课程模块，以便将其推广到其他大学。