Collaborative Research: NCS: FO: Enhancing Episodic Memory through Real-world Integration of Brain Recording and Stimulation with Semantic Alignment of Human and IoT Perception

合作研究：NCS：FO：通过在现实世界中整合大脑记录和刺激以及人类和物联网感知的语义对齐来增强情景记忆

• 批准号: 2124252
• 负责人: Cory Inman
• 金额: $ 50万
• 依托单位: University of Utah
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-09-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2124252&HistoricalAwards=false
• 关键词: Collaborative; Research; NCS; FO; Enhancing

The goal of this project is to build and evaluate a system for exploring how the human brain processes information in everyday real-world environments. The investigators will directly record from the brain while participants navigate the real world, while synchronously recording information about the participant's first-person experience from a set of sensors including cameras, microphones, eye-tracking, and physiological recordings. Neurosurgical participants with a clinically implanted neural recording and stimulation system volunteer for these experiments, providing rare direct human brain recordings as they move around a real-world environment. The rich sensor data captured from the participant's first-person experience will be analyzed in relation to the neural data to infer how changes in patterns of neural activity over time relate to changes in experience. In addition, stimulation will be applied, at safe levels and timed according to "event boundaries" of the participant's experience, to determine whether memories of specific events can be enhanced. The proposed platform that allows for neural recording, direct brain stimulation, and synchronization with external, wearable devices will open an entirely new area of research at the intersection of computer science, engineering, cognition, and clinical neuroscience. These studies will launch and accelerate an emerging and pivotal area of research that will provide therapeutic interventions, proven in the real-world, for participants afflicted with debilitating cognitive disorders. This project will also make substantial contributions to education and outreach, including the development of K-12 classroom modules, interdisciplinary graduate training, outreach to industry partners in the neuromodulation field, and workshops at local Salt Lake City memory care communities.Development of this neural and first-person experience recording system will entail three collaborative research tasks: i) Synchronizing the Human Experience Relative to Neuronal Events: This module will develop a robust framework to record and synchronize neuronal activity along with internet of things (IoT) sensor data representing a broad subset of human sensory channels. The design will be portable such that the human experience can be reasoned about outside of a simulated lab environment. ii) Real-time Semantic Alignment between Human and IoT Perception: Reasoning about the complex relationships between neural biomarkers and the human experience captured by IoT sensing requires more than sensor synchronization. Neural-symbolic approaches that integrate the perception capabilities of deep learning with human logic will be leveraged to reason about the high-level complex spatiotemporal dependencies across a heterogeneous set of sensors. iii) Enhancing Episodic Memories of Real-world Experiences with DBS: Given a proper characterization of neural oscillations associated with event boundaries, the investigators will work to enhance episodic memories of real-world experiences with wireless deep brain stimulation (DBS) devices by directly stimulating the human brain. Under medical supervision, stimulation will be applied to the human amygdala at and between event boundaries in subjects with implanted stimulation devices as they encounter novel, 3D augmented reality objects while navigating a large-scale, real-world environment. Memory will be subsequently tested in laboratory and real-world settings. These three research areas will develop a situational understanding of neuronal activity in the context of human experience. They will further lay the foundation for future research directions in safe and effective stimulation of the brain in response to human experience in the wild.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

这个项目的目标是建立和评估一个系统，用于探索人类大脑如何在日常现实世界环境中处理信息。研究人员将在参与者在现实世界中导航时直接从大脑记录，同时从一组传感器(包括摄像头、麦克风、眼球跟踪和生理记录)同步记录关于参与者的第一人称体验的信息。临床植入神经记录和刺激系统的神经外科参与者自愿参加这些实验，当他们在真实世界环境中移动时，提供罕见的直接人脑记录。从参与者的第一人称体验中捕获的丰富传感器数据将与神经数据进行分析，以推断神经活动模式随时间的变化如何与体验的变化相关。此外，刺激将在安全水平上进行，并根据参与者经验的“事件边界”确定时间，以确定是否可以增强对特定事件的记忆。拟议中的平台允许神经记录、直接大脑刺激，并与外部可穿戴设备同步，将在计算机科学、工程学、认知学和临床神经科学的交叉点开辟一个全新的研究领域。这些研究将启动并加速一个新兴的关键研究领域，为患有衰弱认知障碍的参与者提供经现实世界证明的治疗干预措施。该项目还将对教育和推广做出重大贡献，包括开发K-12课堂模块、跨学科研究生培训、与神经调节领域的行业合作伙伴的扩展，以及在当地盐湖城记忆护理社区的研讨会。开发这个神经和第一人称经验记录系统将需要三项合作研究任务：i)与神经元事件相关的人类经验同步：该模块将开发一个强大的框架，以记录和同步物联网(IoT)传感器数据和代表人类感觉通道的广泛子集的神经元活动。该设计将是便携的，这样人类的体验就可以在模拟的实验室环境之外进行推理。Ii)人类和物联网感知之间的实时语义对齐：关于神经生物标记物和物联网感知捕获的人类体验之间的复杂关系的推理需要的不仅仅是传感器同步。将深度学习的感知能力与人类逻辑相结合的神经符号方法将被用来推理一组不同类型传感器之间的高水平复杂时空依赖关系。Iii)利用DBS增强现实世界经验的情节记忆：鉴于与事件边界相关的神经振荡的适当特征，研究人员将致力于通过直接刺激人脑来增强使用无线脑深部刺激(DBS)设备对现实世界经验的情节记忆。在医学监督下，当受试者在大规模的真实世界环境中导航时，当他们遇到新颖的3D增强现实对象时，将在事件边界处和事件边界之间对人类杏仁核进行刺激。随后将在实验室和真实世界环境中测试记忆。这三个研究领域将在人类经验的背景下发展对神经元活动的情景理解。他们将进一步为未来安全有效地刺激大脑的研究方向奠定基础，以回应人类在野外的经验。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

X-CHAR: A Concept-based Explainable Complex Human Activity Recognition Model

• DOI: 10.1145/3580804
• 发表时间: 2022-03
• 期刊: Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies
• 作者: J. Jeyakumar;Ankur Sarker;L. Garcia;Mani Srivastava