CHS: Small: Collaborative Research: APERTURE: Augmented Reality based Perception-Sensitive Robotic Gesture

CHS：小型：协作研究：APERTURE：基于增强现实的感知敏感机器人手势

• 批准号: 1909864
• 负责人: Thomas Williams
• 金额: $ 25.73万
• 依托单位: Colorado School of Mines
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-09-01 至 2023-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1909864&HistoricalAwards=false
• 关键词: CHS; Small; Collaborative; Research; APERTURE

How can a robot choose the "best" modality for drawing the attention of a human and communicating the needed information? This is the central theme of the project with the assumption that the human team mates of the robots use augmented reality (AR) based visualization. Specifically, this project plans the following three major research activities for enabling robots to communicate with human teammates in a way that is tailored to their teammates' current mental states: exploring how augmented reality technologies (such as the Microsoft Hololens) can be used to provide robots with new ways to communicate about objects, locations, and people in their environments, especially when used together with communication through spoken language; examining how technologies can non-invasively measure different aspects of teammates' mental states, including how much mental workload, perceptual workload, stress, and frustration they are experiencing; and determining how robots can choose the best way to communicate with their human teammates when the two technologies are combined, (for example, through language alone, AR visualizations alone, or both used together), based on those teammates' individual mental states. This system will then be used to test how it might improve the safety and productivity of underground workers, by allowing robots to communicate in a way that is more effective and less cognitively demanding. While the researchers will be investigating the effectiveness of these integrated technologies specifically within underground work environments, the research will also be applicable to a wide variety of areas, including eldercare, urban search-and-rescue, and space robotics, and will have broad scientific impact across both computer science and cognitive science.The above goals will be achieved through APERTURE, a novel framework integrating head-mounted augmented reality displays, a multimodal suite of noninvasive, lightweight, and field-ready physiological sensors (such as functional near-infrared spectroscopy (fNIRS), Electroencaphalography (EEG), Electrodermal Activity, Electrocardiogram (ECG), and Respiration sensors), and unmanned ground robots, within a cognitive robotic architecture. APERTURE will be built by integrating the Distributed Integrated Affect Reflection Cognition (DIARC) architecture with these robotic, augmented reality, and physiological hardware elements. The project will design and evaluate physiological sensing models, augmented reality gestural cues, and machine learning models for selecting between AR gestural cues based on neurophysiological data. The designed machine learning models will classify users' cognitive and affective states from this sensor data, and help the robots understand when and how to communicate based on users' cognitive and affective states. The novel AR approach to deictic gesture will help robots pick out the objects they are referring to through the use of visualizations displayed in their teammates' augmented reality headsets.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

机器人如何选择“最佳”模态来吸引人类的注意力并传达所需的信息？这是该项目的中心主题，假设机器人的人类队友使用基于增强现实（AR）的可视化。具体而言，该项目计划开展以下三项主要研究活动，使机器人能够以适合其队友当前精神状态的方式与人类队友进行交流：探索增强现实技术（如微软Hololens）可以用来为机器人提供新的方式来交流物体，位置和环境中的人，特别是当与口语交流一起使用时;研究技术如何非侵入性地测量队友心理状态的不同方面，包括他们正在经历的心理工作量，感知工作量，压力和挫折;以及确定当这两种技术相结合时，机器人如何选择与人类队友进行交流的最佳方式（例如，仅通过语言，仅通过AR可视化，或两者一起使用），基于这些队友的个人心理状态。然后，该系统将被用于测试如何通过允许机器人以更有效和认知要求更低的方式进行通信来提高地下工人的安全性和生产力。虽然研究人员将调查这些集成技术在地下工作环境中的有效性，但该研究也将适用于各种领域，包括老年人护理，城市搜救和空间机器人，并将在计算机科学和认知科学方面产生广泛的科学影响。一个新的框架，集成了头戴式增强现实显示器，一个多模式套件的非侵入性，重量轻，现场准备的生理传感器，（如功能性近红外光谱（fNIRS）、脑电图（EEG）、皮肤电活动、心电图（ECG）和呼吸传感器）和无人地面机器人，在一个认知机器人架构中。APERTURE将通过将分布式集成情感反射认知（DIARC）架构与这些机器人，增强现实和生理硬件元素集成来构建。 该项目将设计和评估生理感知模型，增强现实手势提示和机器学习模型，用于基于神经生理数据在AR手势提示之间进行选择。设计的机器学习模型将从这些传感器数据中分类用户的认知和情感状态，并帮助机器人了解何时以及如何基于用户的认知和情感状态进行通信。这种新颖的AR指示手势方法将帮助机器人通过使用队友增强现实耳机中显示的可视化效果来挑选它们所指的物体。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Is it Pointless? Modeling and Evaluation of Category Transitions of Spatial Gestures

这是毫无意义的吗？

• DOI: 10.1145/3434074.3447199
• 发表时间: 2021
• 期刊: ACM/IEEE International Conference on Human-Robot Interaction
• 作者: Stogsdill, Adam;Clark, Grace;Ranucci, Aly;Phung, Thao;Williams, Tom
• 通讯作者: Williams, Tom

Robot-generated Mixed Reality Gestures Improve Human-Robot Interaction

机器人生成的混合现实手势改善人机交互

• DOI: 10.1007/978-3-030-90525-5_69
• 发表时间: 2022
• 期刊: International Conference on Social Robotics (Brief Research Reports
• 作者: Tran, Nhan;Grant, Trevor;Phung, Thao;Hirshfield, Leanne;Wickens, Christopher;Williams, Tom
• 通讯作者: Williams, Tom

Crossing Reality: Comparing Physical and Virtual Robot Deixis

穿越现实：比较物理和虚拟机器人指示语

• DOI: 10.1145/3568162.3576972
• 发表时间: 2023
• 期刊: ACM/IEEE International Conference on Human-Robot Interaction
• 作者: Han, Zhao;Zhu, Yifei;Phan, Albert;Garza, Fernando Sandoval;Castro, Amia;Williams, Tom
• 通讯作者: Williams, Tom

Projecting Robot Navigation Paths: Hardware and Software for Projected AR

投影机器人导航路径：投影 AR 的硬件和软件

• DOI: 10.5555/3523760.3523843
• 发表时间: 2022
• 期刊: ACMIEEE International Conference on HumanRobot Interaction
• 作者: Han, Zhao;Parrillo, Jenna;Wilkinson, Alexander;Yanco, Holly A.;Williams, Tom
• 通讯作者: Williams, Tom

Toward Workload-Based Adaptive Automation: The Utility of fNIRS for Measuring Load in Multiple Resources in the Brain

• DOI: 10.1080/10447318.2023.2266242
• 发表时间: 2023-10-21
• 期刊: INTERNATIONAL JOURNAL OF HUMAN-COMPUTER INTERACTION
• 影响因子: 4.7
• 作者: Hirshfield，Leanne M.;Wickens，Christopher;Hayne，Lucas
• 通讯作者: Hayne，Lucas