Bayesian-centric Multimodal Hands-free Computer Interaction Technologies for People with Quadriplegia

与电脑互动对四肢瘫痪的人来说仍然是一个挑战。能够与计算机进行免提交互的辅助技术主要基于眼睛注视、声音和口头控制的输入方式，每种方式都有自己的优点和缺点。然而，这些辅助技术不支持多种输入方式的协同使用，例如使用眼睛注视来快速缩小包含执行口头命令的预期目标的区域。该项目的总体目标是研究、设计和设计智能和协作的多模态免提交互技术，这些技术可以协同地组合来自不同输入模式的输入，以准确预测并根据用户的交互意图采取行动。输入模式的协同整合和用户交互意图的智能推断放大了单个模式的集体优势，同时减轻了它们的弱点。更重要的是，这些技术还将从用户的交互历史中学习用户特定的交互模式，以便个性化预测每个用户的预期操作。总的来说，这个项目将产生的变革性辅助多模式交互系统SeeSayClick，将使四肢瘫痪的人更容易创建和消费数字信息，从而充分参与到这个数字化经济中来。由此产生的这些用户的更高生产力将导致获得更好的教育和就业机会。最后，这个项目将成为培训学生的平台，让他们接触到辅助技术开发和康复工程方面的职业。设想中的SeeSayClick辅助技术的新颖性将是多种交互模式的紧密集成，这些交互模式将协同工作并解决交互中的歧义，从而大大减少交互负担。集成的基础将植根于人机交互的贝叶斯推理方法。这些方法提供了一种原则性的方法，可以结合多个信息源（可能有噪声）来预测用户的预期交互动作，例如将：(1)来自凝视的位置信息与(2)口头命令相结合，以及(3)来自交互上下文的先验知识相结合，以推断预期目标的精确位置，以便选择和执行。通过将先前的交互历史合并到贝叶斯方法中，所提出的集成多种输入模式的方法还将学习用户特定的交互模式，以个性化预测并进一步提高每个用户的预测准确性。除了光标操作和命令执行之外，贝叶斯方法还将与用于文本输入和编辑操作的语言模型耦合在一起。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Interacting with computers remains a challenge for people with quadriplegia. Assistive technologies that enable hands-free interaction with computers are primarily based on eye-gaze, voice, and orally-controlled input modalities, each with its own strengths and weaknesses. However, these assistive technologies do not support collaborative use of multiple input modalities, such as using eye gaze to quickly narrow down the region containing the intended target for executing a spoken command. The overarching goal of the proposed project is to research, design and engineer intelligent and collaborative multimodal hands-free interaction techniques that synergistically combine inputs from different input modalities to accurately predict and act on the user's interaction intent. Synergistic integration of the input modalities and intelligent inferring of the user’s interaction intent amplify the collective strengths of the individual modalities while mitigating their weaknesses. More importantly, these techniques will also learn user-specific interaction patterns from the user’s interaction history for personalizing the prediction of each individual user’s intended action. Overall, the transformative assistive multimodal interaction system, SeeSayClick, that will emerge from this project, will make it far easier for people with quadriplegia to create and consume digital information and thereby fully participate in this digitized economy. The resulting higher productivity of such users will lead to improved access to education and employment opportunities. Lastly, this project will serve as a platform for training students and exposing them to careers in assistive technology development and rehabilitation engineering.The novelty of the envisioned SeeSayClick assistive technology will be the tight integration of multiple interaction modalities that will work together synergistically and resolve ambiguities in interaction, and as a consequence, reduce the interaction burden substantially. The basis for the integration will be rooted in Bayesian inference methods for human computer interaction. These methods provide a principled approach for combining multiple sources of information, possibly noisy, to predict the user's intended interaction action, such as combining: (1) the locational information from gaze with (2) the spoken commands, and (3) prior knowledge from the interaction context to infer the intended target's precise location for selection and execution. By incorporating interaction history as prior into the Bayesian methods, the proposed approach for integrating multiple input modalities will also learn user-specific interaction patterns to personalize the prediction and enhance the prediction accuracy even further for each individual user. Besides cursor operations and command execution, the Bayesian methods will be coupled to a language model for text entry and editing operations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.