NRI: Collaborative Research: Multimodal Brain Computer Interface for Human-Robot Interaction

NRI：协作研究：用于人机交互的多模式脑机接口

• 批准号: 1527747
• 负责人: Peter Allen
• 金额: $ 73.66万
• 依托单位: Columbia University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-05-15 至 2020-04-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1527747&HistoricalAwards=false
• 关键词: NRI; Collaborative; Research; Multimodal; Brain

Human Robot Interaction (HRI) is research that is a key component in making robots part of our everyday life. Current interface modalities such as video, keyboard, tactile, audio, and speech can all contribute to an HRI interface. However, an emerging area is the use of Brain-Computer Interfaces (BCI) for communication and information exchange between humans and robots. BCIs can provide another channel of communication with more direct access to physiological changes in the brain. BCIs vary widely in their capabilities, particularly with respect to spatial resolution, temporal resolution and noise. This project is aimed at exploring the use of multimodal BCIs for HRI. Multimodal BCIs, also referred to as hybrid BCIs (hBCI), have been shown to improve performance over single modality interfaces. This project is focused on using a novel suite of sensors (Electroencephalography (EEG), eye-tracking, pupillary size, computer vision, and functional Near Infrared Spectroscopy (fNIRS)) to improve current HRI systems. Each of these sensing modalities can reinforce and complement each other, and when used together, can address a major shortcoming of current BCIs which is the determination of the user state or situational awareness (SA). SA is a necessary component of any complex interaction between agents, as each agent has its own expectations and assumptions about the environment. Traditional BCI systems have difficulty recognizing state and context, and accordingly can become confusing and unreliable. This project will develop techniques to recognize state from multiple modalities, and will also allow the robot and human to learn about each other's state and expectations using the hBCI we are developing. The goal is to build a usable hBCI for real physical robot environments, with noise, real-time constraints, and added complexity.The technical contributions of this project include:1. Characterization of a novel hBCI interface for visual recognition and labeling tasks with real physical data and environments.2. Integration of fNIRS sensing with EEG and other modalities in human robot interaction tasks. We will test our ability in the temporal domain to determine at what timescale we can correctly classify movement components that would predict a correct (rewarding) trial or non-rewarding/incorrect movement.3. Analysis and validation of the hBCI in complex robotic tele-operation tasks with human subject operators such as open door, grasp object on table, pick up item off floor etc.4. Use of hBCI to characterize human/robot state and create a learning method to recognize state over time.5. Use of augmented reality for HRI decision making.6. Further develop hBCI for tracking cognitive states related to reward, motivation, attention and value.A new class of HRI interfaces will be developed that can expand the ability of humans to work with robots; promote the use and acceptance of robot agent systems in everyday life; expand the use of hBCIs in areas other than robotics for human-machine interaction; further the development of hBCIs as our system will be tapping into reward modulated activity that will be used via reinforcement learning to autonomously update the learning machinery; and bridge the educational divide between Engineering/Computer Science and Neuroscience.

人机交互 (HRI) 是一项研究，它是让机器人成为我们日常生活一部分的关键组成部分。当前的界面模式，例如视频、键盘、触觉、音频和语音，都可以构成 HRI 界面。然而，一个新兴领域是使用脑机接口（BCI）进行人类和机器人之间的通信和信息交换。脑机接口可以提供另一种沟通渠道，更直接地了解大脑的生理变化。 BCI 的功能差异很大，特别是在空间分辨率、时间分辨率和噪声方面。该项目旨在探索多模式 BCI 在 HRI 中的使用。多模态 BCI，也称为混合 BCI (hBCI)，已被证明可以比单模态接口提高性能。该项目的重点是使用一套新型传感器（脑电图 (EEG)、眼球追踪、瞳孔大小、计算机视觉和功能性近红外光谱 (fNIRS)）来改进当前的 HRI 系统。这些传感方式中的每一种都可以相互增强和补充，并且当一起使用时，可以解决当前 BCI 的主要缺点，即用户状态或态势感知 (SA) 的确定。 SA 是智能体之间任何复杂交互的必要组成部分，因为每个智能体对环境都有自己的期望和假设。传统的 BCI 系统难以识别状态和上下文，因此可能会变得混乱且不可靠。该项目将开发从多种模式识别状态的技术，并且还允许机器人和人类使用我们正在开发的 hBCI 了解彼此的状态和期望。目标是为真实的物理机器人环境构建一个可用的 hBCI，具有噪声、实时约束和增加的复杂性。该项目的技术贡献包括： 1．一种新颖的 hBCI 接口的表征，用于具有真实物理数据和环境的视觉识别和标记任务。 2．在人机交互任务中将 fNIRS 传感与脑电图和其他模式集成。我们将测试我们在时域中的能力，以确定在什么时间尺度上我们可以正确分类运动成分，从而预测正确的（奖励）尝试或非奖励/不正确的运动。3。分析和验证 hBCI 在复杂机器人远程操作任务中与人类操作员的关系，例如开门、抓取桌子上的物体、拾取地板上的物品等4。使用 hBCI 来表征人类/机器人状态并创建一种学习方法来识别随时间变化的状态。5。使用增强现实进行 HRI 决策。6．进一步开发hBCI，用于跟踪与奖励、动机、注意力和价值相关的认知状态。将开发一类新的HRI接口，可以扩展人类与机器人合作的能力；促进机器人代理系统在日常生活中的使用和接受；扩大 hBCIs 在人机交互机器人以外的领域的使用；进一步发展 hBCIs，因为我们的系统将利用奖励调节活动，通过强化学习来自主更新学习机制；弥合工程/计算机科学和神经科学之间的教育鸿沟。