NRI: Collaborative Research: Multimodal Brain Computer Interface for Human-Robot Interaction

人机交互（HRI）研究是使机器人成为我们日常生活一部分的关键组成部分。当前的接口形式，如视频、键盘、触觉、音频和语音都可以用于HRI接口。然而，一个新兴的领域是使用脑机接口（BCI）进行人与机器人之间的通信和信息交换。脑机接口可以提供另一种更直接接触大脑生理变化的沟通渠道。脑机接口的能力差别很大，特别是在空间分辨率、时间分辨率和噪声方面。本项目旨在探索多模式脑机接口在HRI中的应用。多模态bci，也被称为混合bci (hBCI)，已经被证明可以提高单模态接口的性能。该项目的重点是使用一套新颖的传感器（脑电图（EEG）、眼动追踪、瞳孔大小、计算机视觉和功能性近红外光谱（fNIRS））来改进当前的HRI系统。这些传感模式可以相互加强和补充，当一起使用时，可以解决当前bci的一个主要缺点，即确定用户状态或态势感知（SA）。情景识别是代理之间任何复杂交互的必要组成部分，因为每个代理对环境都有自己的期望和假设。传统的BCI系统难以识别状态和上下文，因此可能变得混乱和不可靠。该项目将开发从多种模式识别状态的技术，并将允许机器人和人类使用我们正在开发的hBCI了解彼此的状态和期望。目标是为真实的物理机器人环境构建一个可用的hBCI，具有噪声，实时限制和增加的复杂性。本项目的技术贡献包括：1。具有真实物理数据和环境的视觉识别和标记任务的新型hBCI接口的表征。人机交互任务中fNIRS传感与脑电图等模式的集成。我们将测试我们在时间域的能力，以确定在什么时间尺度上我们可以正确地分类运动成分，这些运动成分可以预测正确的（奖励）试验或非奖励/错误的运动。3 .基于人机交互的复杂机器人远程操作任务（如开门、抓取桌上物体、拾取地上物品等）的hBCI分析与验证4 .使用hBCI来表征人/机器人的状态，并创建一种学习方法来识别状态。增强现实技术在人力资源调研决策中的应用进一步发展hBCI以跟踪与奖励、动机、注意力和价值相关的认知状态。将开发一类新的HRI接口，可以扩展人类与机器人合作的能力；促进机器人代理系统在日常生活中的使用和接受；在人机交互机器人以外的领域扩大hbci的使用；进一步发展hBCIs，因为我们的系统将利用奖励调节活动，通过强化学习来自主更新学习机制；并弥合工程学和神经科学之间的教育鸿沟。

Human Robot Interaction (HRI) is research that is a key component in making robots part of our everyday life. Current interface modalities such as video, keyboard, tactile, audio, and speech can all contribute to an HRI interface. However, an emerging area is the use of Brain-Computer Interfaces (BCI) for communication and information exchange between humans and robots. BCIs can provide another channel of communication with more direct access to physiological changes in the brain. BCIs vary widely in their capabilities, particularly with respect to spatial resolution, temporal resolution and noise. This project is aimed at exploring the use of multimodal BCIs for HRI. Multimodal BCIs, also referred to as hybrid BCIs (hBCI), have been shown to improve performance over single modality interfaces. This project is focused on using a novel suite of sensors (Electroencephalography (EEG), eye-tracking, pupillary size, computer vision, and functional Near Infrared Spectroscopy (fNIRS)) to improve current HRI systems. Each of these sensing modalities can reinforce and complement each other, and when used together, can address a major shortcoming of current BCIs which is the determination of the user state or situational awareness (SA). SA is a necessary component of any complex interaction between agents, as each agent has its own expectations and assumptions about the environment. Traditional BCI systems have difficulty recognizing state and context, and accordingly can become confusing and unreliable. This project will develop techniques to recognize state from multiple modalities, and will also allow the robot and human to learn about each other's state and expectations using the hBCI we are developing. The goal is to build a usable hBCI for real physical robot environments, with noise, real-time constraints, and added complexity.The technical contributions of this project include:1. Characterization of a novel hBCI interface for visual recognition and labeling tasks with real physical data and environments.2. Integration of fNIRS sensing with EEG and other modalities in human robot interaction tasks. We will test our ability in the temporal domain to determine at what timescale we can correctly classify movement components that would predict a correct (rewarding) trial or non-rewarding/incorrect movement.3. Analysis and validation of the hBCI in complex robotic tele-operation tasks with human subject operators such as open door, grasp object on table, pick up item off floor etc.4. Use of hBCI to characterize human/robot state and create a learning method to recognize state over time.5. Use of augmented reality for HRI decision making.6. Further develop hBCI for tracking cognitive states related to reward, motivation, attention and value.A new class of HRI interfaces will be developed that can expand the ability of humans to work with robots; promote the use and acceptance of robot agent systems in everyday life; expand the use of hBCIs in areas other than robotics for human-machine interaction; further the development of hBCIs as our system will be tapping into reward modulated activity that will be used via reinforcement learning to autonomously update the learning machinery; and bridge the educational divide between Engineering and Neuroscience.