CAREER: HCC: Microgesture and Multimodal Interaction Techniques for Augmented Reality

增强现实（AR）眼镜通过允许用户在真实环境中操作虚拟物体，正在改变工作场所。从办公室到制造业再到服务业，增强现实技术带来了深远的好处，但通过增强现实眼镜改善用户与虚拟世界的互动方式对于这一转变至关重要。AR系统必须被设计成能够理解人类交流的微妙和复杂性，这是一种广泛的模式组合——手势、语音、文本、凝视等等。目前的AR头戴式系统无法处理多种通信模式，而且它们只能识别粗糙的手势，这些手势对用户来说既尴尬又费力。减少疲劳和促进直观交互的一种方法是让系统识别和处理微手势。在这个项目中，研究这些简单的小手指动作，类似于控制汽车收音机音量的动作，以建立使人类- ar交互系统更加直观和广泛可访问所需的基础知识。减少与AR技术互动的障碍将使人们能够专注于他们的任务，而不是技术本身。这项研究影响了气象、金融、医疗保健和科学可视化等领域，促进了丰富的数据探索，并改善了数据理解和决策。在大规模开放在线课程（MOOCs）中实施这项研究将有利于独立学习的学生和教育机构的学生，允许任何对这一主题感兴趣的人访问。本项目通过研究直观微手势交互技术和附加通信方式，推进AR多模态交互研究。这些研究的拟议研究计划包括与语音、6自由度（6DoF）笔、控制器和空中手势相结合的输入。还将评估人机交互。这些研究的结果将回答一个重要的问题，“微手势和多模态交互在AR中应该是什么样子？”，从而形成了多模态交互的统一框架，并创建了多模态交互工具包（MITK）。沉浸式分析（即3D数据可视化）将被用作执行系统验证的领域。此外，研究人员将评估并将研究结果从AR扩展到虚拟现实（VR）。本研究通过以下方式促进了AR和VR交互的发展：(1)研究和开发用于选择、操作和导航的微手势；(2)设计、开发和评估新的单模态和多模态交互技术，包括虚拟部件；(3)扩展微手势的基本知识及其在联合任务中的应用。研究小组将开发一个新的多模态交互教科书和一个使用沉浸式分析的多模态交互开放在线课程。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Augmented reality (AR) glasses are transforming the workplace by allowing users to manipulate virtual objects in a real environment. From offices to manufacturing to service, AR technology offers far-reaching benefits, but improving how users interact with virtual worlds through AR glasses is essential to this transformation. AR systems must be designed to understand the subtleties and complexities of human communication, which come in a broad combination of modes—gestures, speech, text, gaze, and more. Current AR head-mounted systems cannot process multiple communication modes and they recognize only coarse gestures that are awkward and tiring for users to produce. One approach to reducing fatigue and promoting intuitive interaction is for the system to recognize and process microgestures. In this project, these simple, small finger movements, akin to those needed to control the volume on a car radio, are studied to build foundational knowledge needed to make human-AR interaction systems more intuitive and broadly accessible. Reducing the barriers to interaction with AR technology will allow people to concentrate on their tasks rather than the technology itself. This research impacts fields such as meteorology, finance, healthcare, and scientific visualization, facilitating rich data exploration, and improving data understanding and decision making. Implementing this research in massive open online courses (MOOCs) will benefit students learning independently and in education institutions, allowing access to anyone interested in this topic.This project advances AR multimodal interaction research by studying intuitive microgesture interaction techniques and additional communication modalities. The proposed research plan for these studies involves inputs combined with speech, a 6 degrees-of-freedom (6DoF) pen, a controller, and midair gestures. Bimanual interaction will also be evaluated. The results of these studies will answer an important question, “What should microgestures and multimodal interaction look like in AR?”, leading to a unified framework of multimodal interactions and the creation of a Multimodal Interaction ToolKit (MITK). Immersive analytics (that is, 3D data visualizations) will be used as a domain for performing system validations. Further, the investigator will evaluate and extend the findings from AR to virtual reality (VR). This research advances the state of the art in AR and VR interaction by: (1) researching and developing microgestures for selection, manipulation, and navigation; (2) designing, developing, and evaluating novel unimodal and multimodal interaction techniques, including virtual widgets; and (3) expanding fundamental knowledge of microgestures and their use in joint tasks. The research team will develop a new multimodal interaction textbook and an open online course on multimodal interaction using immersive analytics.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.