CAREER: HCC: Microgesture and Multimodal Interaction Techniques for Augmented Reality

增强现实（AR）眼镜通过允许用户在真实的环境中操纵虚拟对象，正在改变工作场所。从办公室到制造业再到服务业，AR技术都能带来深远的好处，但改善用户通过AR眼镜与虚拟世界互动的方式对这一转变至关重要。增强现实系统必须被设计成能够理解人类交流的微妙性和复杂性，而人类交流的方式包括手势、语音、文本、凝视等等。目前的AR头戴式系统无法处理多种通信模式，并且它们只能识别用户产生的笨拙和疲劳的粗糙手势。减少疲劳和促进直观交互的一种方法是系统识别和处理微手势。在这个项目中，研究这些简单的小手指运动，类似于控制汽车收音机音量所需的运动，以建立使人类-AR交互系统更直观和更广泛访问所需的基础知识。减少与AR技术互动的障碍将使人们能够专注于他们的任务，而不是技术本身。这项研究影响了气象学，金融，医疗保健和科学可视化等领域，促进了丰富的数据探索，并提高了数据理解和决策。在大规模开放式在线课程（MOOC）中实施这项研究将有利于学生独立学习和在教育机构学习，允许任何对该主题感兴趣的人访问。该项目通过研究直观的微手势交互技术和其他通信方式来推进AR多模态交互研究。这些研究的拟议研究计划涉及与语音相结合的输入，6自由度（6DoF）笔，控制器和空中手势。还将评价双手交互。这些研究的结果将回答一个重要的问题，“微手势和多模式交互在AR中应该是什么样子？”，导致多模态交互的统一框架和多模态交互工具包（MITK）的创建。沉浸式分析（即3D数据可视化）将被用作执行系统验证的域。此外，研究人员将评估并将研究结果从AR扩展到虚拟现实（VR）。这项研究通过以下方式推进了AR和VR交互的最新发展：（1）研究和开发用于选择，操作和导航的微手势;（2）设计，开发和评估新的单峰和多峰交互技术，包括虚拟部件;（3）扩展微手势的基础知识及其在联合任务中的使用。该研究团队将开发一本新的多模态交互教科书和一门关于使用沉浸式分析的多模态交互的开放式在线课程。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Augmented reality (AR) glasses are transforming the workplace by allowing users to manipulate virtual objects in a real environment. From offices to manufacturing to service, AR technology offers far-reaching benefits, but improving how users interact with virtual worlds through AR glasses is essential to this transformation. AR systems must be designed to understand the subtleties and complexities of human communication, which come in a broad combination of modes—gestures, speech, text, gaze, and more. Current AR head-mounted systems cannot process multiple communication modes and they recognize only coarse gestures that are awkward and tiring for users to produce. One approach to reducing fatigue and promoting intuitive interaction is for the system to recognize and process microgestures. In this project, these simple, small finger movements, akin to those needed to control the volume on a car radio, are studied to build foundational knowledge needed to make human-AR interaction systems more intuitive and broadly accessible. Reducing the barriers to interaction with AR technology will allow people to concentrate on their tasks rather than the technology itself. This research impacts fields such as meteorology, finance, healthcare, and scientific visualization, facilitating rich data exploration, and improving data understanding and decision making. Implementing this research in massive open online courses (MOOCs) will benefit students learning independently and in education institutions, allowing access to anyone interested in this topic.This project advances AR multimodal interaction research by studying intuitive microgesture interaction techniques and additional communication modalities. The proposed research plan for these studies involves inputs combined with speech, a 6 degrees-of-freedom (6DoF) pen, a controller, and midair gestures. Bimanual interaction will also be evaluated. The results of these studies will answer an important question, “What should microgestures and multimodal interaction look like in AR?”, leading to a unified framework of multimodal interactions and the creation of a Multimodal Interaction ToolKit (MITK). Immersive analytics (that is, 3D data visualizations) will be used as a domain for performing system validations. Further, the investigator will evaluate and extend the findings from AR to virtual reality (VR). This research advances the state of the art in AR and VR interaction by: (1) researching and developing microgestures for selection, manipulation, and navigation; (2) designing, developing, and evaluating novel unimodal and multimodal interaction techniques, including virtual widgets; and (3) expanding fundamental knowledge of microgestures and their use in joint tasks. The research team will develop a new multimodal interaction textbook and an open online course on multimodal interaction using immersive analytics.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.