NRI: Collaborative Research: Multimodal Brain Computer Interface for Human-Robot Interaction

人机交互（HRI）是使机器人成为我们日常生活一部分的关键组成部分。当前的接口形式，如视频，键盘，触觉，音频和语音都可以有助于HRI接口。然而，一个新兴的领域是使用脑机接口（BCI）进行人类和机器人之间的通信和信息交换。脑机接口可以提供另一种沟通渠道，更直接地了解大脑的生理变化。脑机接口在其能力方面变化很大，特别是在空间分辨率、时间分辨率和噪声方面。该项目旨在探索多模式脑机接口在HRI中的应用。多模态脑机接口（也称为混合脑机接口（hBCI））已被证明可以提高单模态接口的性能。该项目的重点是使用一套新的传感器（脑电图（EEG），眼睛跟踪，瞳孔大小，计算机视觉和功能性近红外光谱（fNIRS））来改善当前的HRI系统。这些感测模态中的每一个都可以相互加强和补充，并且当一起使用时，可以解决当前BCI的主要缺点，即确定用户状态或情景感知（SA）。SA是代理之间任何复杂交互的必要组成部分，因为每个代理都有自己对环境的期望和假设。传统的BCI系统难以识别状态和上下文，因此可能变得混乱和不可靠。该项目将开发从多种模式中识别状态的技术，并允许机器人和人类使用我们正在开发的hBCI来了解彼此的状态和期望。目标是为真实的物理机器人环境构建一个可用的hBCI，具有噪声，实时约束和增加的复杂性。表征一种新的hBCI接口的视觉识别和标记任务与真实的物理数据和环境. fNIRS传感与EEG和其他模态在人类机器人交互任务中的集成。我们将在时间域中测试我们的能力，以确定在什么时间尺度上我们可以正确地对运动成分进行分类，从而预测正确（奖励）试验或非奖励/不正确的运动。分析和验证了hBCI在复杂的机器人遥操作任务中的应用，如开门，抓取桌子上的物体，捡起地板上的物品等。使用hBCI来表征人类/机器人状态，并创建一种学习方法来识别随时间变化的状态。使用增强现实进行HRI决策。进一步开发hBCI，用于跟踪与奖励、动机、注意力和价值相关的认知状态。将开发一类新的HRI接口，以扩展人类与机器人合作的能力;促进机器人代理系统在日常生活中的使用和接受;扩大hBCI在机器人以外的人机交互领域的使用;进一步发展hBCI，因为我们的系统将利用奖励调制活动，通过强化学习来自主更新学习机制;并弥合工程和神经科学之间的教育鸿沟。

Human Robot Interaction (HRI) is research that is a key component in making robots part of our everyday life. Current interface modalities such as video, keyboard, tactile, audio, and speech can all contribute to an HRI interface. However, an emerging area is the use of Brain-Computer Interfaces (BCI) for communication and information exchange between humans and robots. BCIs can provide another channel of communication with more direct access to physiological changes in the brain. BCIs vary widely in their capabilities, particularly with respect to spatial resolution, temporal resolution and noise. This project is aimed at exploring the use of multimodal BCIs for HRI. Multimodal BCIs, also referred to as hybrid BCIs (hBCI), have been shown to improve performance over single modality interfaces. This project is focused on using a novel suite of sensors (Electroencephalography (EEG), eye-tracking, pupillary size, computer vision, and functional Near Infrared Spectroscopy (fNIRS)) to improve current HRI systems. Each of these sensing modalities can reinforce and complement each other, and when used together, can address a major shortcoming of current BCIs which is the determination of the user state or situational awareness (SA). SA is a necessary component of any complex interaction between agents, as each agent has its own expectations and assumptions about the environment. Traditional BCI systems have difficulty recognizing state and context, and accordingly can become confusing and unreliable. This project will develop techniques to recognize state from multiple modalities, and will also allow the robot and human to learn about each other's state and expectations using the hBCI we are developing. The goal is to build a usable hBCI for real physical robot environments, with noise, real-time constraints, and added complexity.The technical contributions of this project include:1. Characterization of a novel hBCI interface for visual recognition and labeling tasks with real physical data and environments.2. Integration of fNIRS sensing with EEG and other modalities in human robot interaction tasks. We will test our ability in the temporal domain to determine at what timescale we can correctly classify movement components that would predict a correct (rewarding) trial or non-rewarding/incorrect movement.3. Analysis and validation of the hBCI in complex robotic tele-operation tasks with human subject operators such as open door, grasp object on table, pick up item off floor etc.4. Use of hBCI to characterize human/robot state and create a learning method to recognize state over time.5. Use of augmented reality for HRI decision making.6. Further develop hBCI for tracking cognitive states related to reward, motivation, attention and value.A new class of HRI interfaces will be developed that can expand the ability of humans to work with robots; promote the use and acceptance of robot agent systems in everyday life; expand the use of hBCIs in areas other than robotics for human-machine interaction; further the development of hBCIs as our system will be tapping into reward modulated activity that will be used via reinforcement learning to autonomously update the learning machinery; and bridge the educational divide between Engineering and Neuroscience.