CHS: SMALL: Collaborative Research: APERTURE: Augmented Reality and Physio-Enhanced Robotic Gesture

机器人如何选择“最佳”方式来吸引人类的注意并传达所需的信息？这是该项目的中心主题，假设机器人的人类团队成员使用基于增强现实（AR）的可视化。具体而言，该项目计划进行以下三项主要研究活动，以使机器人能够以适合其队友当前心理状态的方式与人类队友进行交流：探索如何使用增强现实技术（如微软Hololens）为机器人提供关于其环境中的物体，位置和人的交流新方法，特别是当与口语交流一起使用时；研究技术如何能够非侵入性地测量队友心理状态的不同方面，包括他们正在经历的心理工作量、感知工作量、压力和沮丧程度；并确定当两种技术结合使用时，机器人如何选择与人类队友沟通的最佳方式（例如，单独通过语言，单独通过AR可视化，或两者同时使用），基于这些队友的个人心理状态。然后，这个系统将被用来测试它如何通过让机器人以一种更有效、更少认知要求的方式进行交流，来提高地下工人的安全和生产力。虽然研究人员将专门研究这些集成技术在地下工作环境中的有效性，但这项研究也将适用于各种领域，包括老年人护理、城市搜救和空间机器人，并将在计算机科学和认知科学领域产生广泛的科学影响。上述目标将通过APERTURE实现。APERTURE是一种新型框架，集成了头戴式增强现实显示器、一套多模态无创、轻量级和现场就绪的生理传感器（如功能性近红外光谱（fNIRS）、脑电图（EEG）、皮肤电活动、心电图（ECG）和呼吸传感器）以及认知机器人架构中的无人地面机器人。APERTURE将通过集成分布式综合影响反射认知（DIARC）架构与这些机器人、增强现实和生理硬件元素来构建。该项目将设计和评估生理传感模型、增强现实手势线索，以及基于神经生理数据在AR手势线索之间进行选择的机器学习模型。设计的机器学习模型将从这些传感器数据中对用户的认知和情感状态进行分类，并帮助机器人了解何时以及如何基于用户的认知和情感状态进行交流。这种新颖的AR指示手势方法将帮助机器人通过使用队友的增强现实耳机显示的可视化来挑选他们所指的物体。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

How can a robot choose the "best" modality for drawing the attention of a human and communicating the needed information? This is the central theme of the project with the assumption that the human team mates of the robots use augmented reality (AR) based visualization. Specifically, this project plans the following three major research activities for enabling robots to communicate with human teammates in a way that is tailored to their teammates' current mental states: exploring how augmented reality technologies (such as the Microsoft Hololens) can be used to provide robots with new ways to communicate about objects, locations, and people in their environments, especially when used together with communication through spoken language; examining how technologies can non-invasively measure different aspects of teammates' mental states, including how much mental workload, perceptual workload, stress, and frustration they are experiencing; and determining how robots can choose the best way to communicate with their human teammates when the two technologies are combined, (for example, through language alone, AR visualizations alone, or both used together), based on those teammates' individual mental states. This system will then be used to test how it might improve the safety and productivity of underground workers, by allowing robots to communicate in a way that is more effective and less cognitively demanding. While the researchers will be investigating the effectiveness of these integrated technologies specifically within underground work environments, the research will also be applicable to a wide variety of areas, including eldercare, urban search-and-rescue, and space robotics, and will have broad scientific impact across both computer science and cognitive science.The above goals will be achieved through APERTURE, a novel framework integrating head-mounted augmented reality displays, a multimodal suite of noninvasive, lightweight, and field-ready physiological sensors (such as functional near-infrared spectroscopy (fNIRS), Electroencaphalography (EEG), Electrodermal Activity, Electrocardiogram (ECG), and Respiration sensors), and unmanned ground robots, within a cognitive robotic architecture. APERTURE will be built by integrating the Distributed Integrated Affect Reflection Cognition (DIARC) architecture with these robotic, augmented reality, and physiological hardware elements. The project will design and evaluate physiological sensing models, augmented reality gestural cues, and machine learning models for selecting between AR gestural cues based on neurophysiological data. The designed machine learning models will classify users' cognitive and affective states from this sensor data, and help the robots understand when and how to communicate based on users' cognitive and affective states. The novel AR approach to deictic gesture will help robots pick out the objects they are referring to through the use of visualizations displayed in their teammates' augmented reality headsets.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.