NRI: Collaborative Research: Multimodal Brain Computer Interface for Human-Robot Interaction

人类-机器人交互(HRI)是使机器人融入我们日常生活的关键组成部分。当前的接口形式，如视频、键盘、触觉、音频和语音，都可以构成HRI接口。然而，一个新兴的领域是使用脑机接口(BCI)来进行人类和机器人之间的通信和信息交换。脑机接口可以提供另一种交流渠道，更直接地获取大脑的生理变化。BCI的能力差异很大，特别是在空间分辨率、时间分辨率和噪声方面。该项目的目的是探索多模式BCI在人力资源倡议中的使用。多模式BCI，也被称为混合BCI(HBCI)，已被证明可以提高单模式接口的性能。该项目的重点是使用一套新的传感器(脑电、眼球跟踪、瞳孔大小、计算机视觉和功能近红外光谱)来改进现有的HRI系统。这些感知模式中的每一种都可以相互加强和补充，当它们一起使用时，可以解决当前BCI的一个主要缺点，即确定用户状态或情景感知(SA)。SA是代理之间任何复杂交互的必要组成部分，因为每个代理都有自己对环境的期望和假设。传统的BCI系统很难识别状态和上下文，因此可能会变得混乱和不可靠。该项目将开发从多个模式识别状态的技术，并将允许机器人和人类使用我们正在开发的hBCI了解彼此的状态和期望。该项目的技术贡献包括：1.描述了一种用于视觉识别和使用真实物理数据和环境的标记任务的新型hBCI接口。在人类机器人交互任务中，将fNIRS传感与脑电和其他模式相结合。我们将在时间域中测试我们的能力，以确定我们在什么时间尺度上可以正确地对预测正确(奖励)试验或非奖励/不正确运动的运动成分进行分类。分析和验证在复杂的机器人遥操作任务中的hBCI，包括开门、抓桌子上的物体、从地上捡起物品等人类主体操作员。使用hBCI来表征人/机器人的状态，并创建一种学习方法来识别随时间推移的状态。使用增强现实技术进行人力资源信息系统决策。进一步开发hBCI跟踪与奖励、动机、注意力和价值相关的认知状态。将开发一类新的HRI界面，可以扩展人类与机器人合作的能力；促进机器人代理系统在日常生活中的使用和接受；扩大hBCI在机器人学以外的领域用于人机交互；进一步开发hBCI，因为我们的系统将利用奖励调节活动，通过强化学习来自主更新学习机器；并弥合工程学和神经科学之间的教育鸿沟。

Human Robot Interaction (HRI) is research that is a key component in making robots part of our everyday life. Current interface modalities such as video, keyboard, tactile, audio, and speech can all contribute to an HRI interface. However, an emerging area is the use of Brain-Computer Interfaces (BCI) for communication and information exchange between humans and robots. BCIs can provide another channel of communication with more direct access to physiological changes in the brain. BCIs vary widely in their capabilities, particularly with respect to spatial resolution, temporal resolution and noise. This project is aimed at exploring the use of multimodal BCIs for HRI. Multimodal BCIs, also referred to as hybrid BCIs (hBCI), have been shown to improve performance over single modality interfaces. This project is focused on using a novel suite of sensors (Electroencephalography (EEG), eye-tracking, pupillary size, computer vision, and functional Near Infrared Spectroscopy (fNIRS)) to improve current HRI systems. Each of these sensing modalities can reinforce and complement each other, and when used together, can address a major shortcoming of current BCIs which is the determination of the user state or situational awareness (SA). SA is a necessary component of any complex interaction between agents, as each agent has its own expectations and assumptions about the environment. Traditional BCI systems have difficulty recognizing state and context, and accordingly can become confusing and unreliable. This project will develop techniques to recognize state from multiple modalities, and will also allow the robot and human to learn about each other's state and expectations using the hBCI we are developing. The goal is to build a usable hBCI for real physical robot environments, with noise, real-time constraints, and added complexity.The technical contributions of this project include:1. Characterization of a novel hBCI interface for visual recognition and labeling tasks with real physical data and environments.2. Integration of fNIRS sensing with EEG and other modalities in human robot interaction tasks. We will test our ability in the temporal domain to determine at what timescale we can correctly classify movement components that would predict a correct (rewarding) trial or non-rewarding/incorrect movement.3. Analysis and validation of the hBCI in complex robotic tele-operation tasks with human subject operators such as open door, grasp object on table, pick up item off floor etc.4. Use of hBCI to characterize human/robot state and create a learning method to recognize state over time.5. Use of augmented reality for HRI decision making.6. Further develop hBCI for tracking cognitive states related to reward, motivation, attention and value.A new class of HRI interfaces will be developed that can expand the ability of humans to work with robots; promote the use and acceptance of robot agent systems in everyday life; expand the use of hBCIs in areas other than robotics for human-machine interaction; further the development of hBCIs as our system will be tapping into reward modulated activity that will be used via reinforcement learning to autonomously update the learning machinery; and bridge the educational divide between Engineering and Neuroscience.