ONR-15-FOA-0011 MURI Topic #3 - Closed-Loop Multisensory Brain-Computer Interface for Enhanced Decision Accuracy

我们跨学科努力的目标是开发新的方法来模拟多感官处理和决策背后的多模态神经活动，并使用这些方法来设计闭环自适应算法，以优化利用多感官数据进行脑机通信。我们的动机是观察到，一个下马的士兵或一个坦克司机通常在时间紧迫和紧张的条件下做出决定，这是基于在混乱和分散注意力的环境中呈现的多种多感官信息。我们设想了一个闭环脑机接口（BCI）架构，以提高决策准确性。该架构将收集多模态神经、生理和行为数据，解码精神状态，如注意力取向和情境感知，并使用解码状态作为反馈，自适应地改变提供给受试者的多感官线索，从而闭合环路。为了实现这样一个架构，我们将在四个方面取得基本进展，构成我们的研究重点：(1)建模多感觉整合，注意和决策，以及相关的神经机制；(2)高维多模态数据融合的机器学习算法；(3)脑机接口在线运行过程中神经和行为模型的自适应跟踪；(4)多感官信号自适应脑机接口控制，优化性能。我们组建了一个多学科团队，拥有跨越工程、计算机科学和神经科学的专业知识。我们将采用一种完全集成的方法，通过结合罕见的最先进的实验能力和新颖的计算模型来解决这些挑战。在啮齿动物、猴子和人类中进行的补充实验将收集多模式数据，以研究和模拟多感觉整合、注意力和决策，并为提高决策准确性制作脑机接口原型。我们的建模工作将跨越贝叶斯推理、随机控制、自适应信号处理和机器学习，以开发：新的大脑机制的贝叶斯和控制理论模型；多模态数据的新随机模型及其自适应推理算法基于用户认知状态反馈的多感官线索自适应随机控制器。

The goals of our interdisciplinary effort are to develop new methodologies for modeling multimodal neural activity underlying multisensory processing and decision making, and to use those methodologies to design closed-loop adaptive algorithms for optimized exploitation of multisensory data for brain-computer communication. We are motivated by the observation that a dismounted soldier or a tank driver routinely makes decisions in time-pressured and stressful conditions based on a multiplicity of multisensory information presented in cluttered and distracting environments. We envision a closed-loop brain-computer interface (BCI) architecture for enhancing decision accuracy. The architecture will collect multimodal neural, physiological, and behavioral data, decode mental states such as attention orientation and situational awareness, and use the decoded states as feedback to adaptively change the multisensory cues provided to the subject, thus closing the loop. To realize such an architecture we will make fundamental advances on four fronts, constituting our research Thrusts: (1) modeling multisensory integration, attention, and decision making, and the associated neural mechanisms; (2) machine-learning algorithms for high-dimensional multimodal data fusion; (3) adaptive tracking of the neural and behavioral models during online operation of the BCI; and (4) adaptive BCI control of multisensory cues for optimized performance. We have assembled a multidisciplinary team with expertise spanning engineering, computer science, and neuroscience. We will take a fully integrated approach to address these challenges by combining rare state-of-the-art experimental capabilities with novel computational modeling. Complementary experiments in rodents, monkeys, and humans will collect multimodal data to study and model multisensory integration, attention, and decision making, and to prototype a BCI for enhanced decision accuracy. Our modeling efforts will span Bayesian inference, stochastic control, adaptive signal processing, and machine learning to develop: novel Bayesian and control-theoretic models of the brain mechanisms; new stochastic models of multimodal data and adaptive inference algorithms for this data; and novel adaptive stochastic controllers of multisensory cues based on the feedback of users' cognitive state.