Collaborative Research: NCS: FO: Enhancing Episodic Memory through Real-world Integration of Brain Recording and Stimulation with Semantic Alignment of Human and IoT Perception

该项目的目标是建立和评估一个系统，用于探索人类大脑如何在日常现实环境中处理信息。当参与者在现实世界中穿行时，研究人员将直接记录参与者的大脑活动，同时通过一组传感器同步记录参与者的第一人称体验信息，这些传感器包括摄像头、麦克风、眼球追踪和生理记录。神经外科手术参与者在临床植入了神经记录和刺激系统，自愿参加这些实验，当他们在现实世界环境中移动时，提供罕见的直接人脑记录。从参与者的第一人称体验中获取的丰富传感器数据将与神经数据进行分析，以推断随着时间的推移，神经活动模式的变化与经验的变化之间的关系。此外，将根据参与者经验的“事件边界”在安全水平和时间范围内施加刺激，以确定是否可以增强对特定事件的记忆。该提议的平台允许神经记录，直接大脑刺激，并与外部可穿戴设备同步，将在计算机科学，工程，认知和临床神经科学的交叉领域开辟一个全新的研究领域。这些研究将启动并加速一个新兴的关键研究领域，为患有衰弱性认知障碍的参与者提供治疗干预措施，并在现实世界中得到证实。该项目还将为教育和推广做出重大贡献，包括开发K-12课堂模块，跨学科研究生培训，向神经调节领域的行业合作伙伴推广，以及在盐湖城当地的记忆护理社区举办讲习班。这种神经和第一人称体验记录系统的开发将需要三个合作研究任务：i)同步相对于神经元事件的人类经验：该模块将开发一个强大的框架来记录和同步神经元活动以及代表人类感觉通道的广泛子集的物联网（IoT）传感器数据。这种设计将是便携式的，这样人类的经验就可以在模拟的实验室环境之外进行推理。ii)人与物联网感知之间的实时语义对齐：对神经生物标志物与物联网感知捕获的人类经验之间复杂关系的推理需要的不仅仅是传感器同步。将深度学习的感知能力与人类逻辑相结合的神经符号方法将被用于推断跨异构传感器集的高级复杂时空依赖关系。iii)通过DBS增强现实世界经验的情景记忆：考虑到与事件边界相关的神经振荡的适当特征，研究人员将通过直接刺激人脑的无线深部脑刺激（DBS）设备来增强现实世界经验的情景记忆。在医学监督下，当受试者在大规模的现实世界环境中遇到新的3D增强现实物体时，将在植入刺激装置的情况下，在事件边界处和事件边界之间对人类杏仁核进行刺激。随后将在实验室和实际环境中对记忆进行测试。这三个研究领域将在人类经验的背景下发展对神经元活动的情境理解。它们将进一步为未来安全有效地刺激大脑响应人类野外经验的研究方向奠定基础。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The goal of this project is to build and evaluate a system for exploring how the human brain processes information in everyday real-world environments. The investigators will directly record from the brain while participants navigate the real world, while synchronously recording information about the participant's first-person experience from a set of sensors including cameras, microphones, eye-tracking, and physiological recordings. Neurosurgical participants with a clinically implanted neural recording and stimulation system volunteer for these experiments, providing rare direct human brain recordings as they move around a real-world environment. The rich sensor data captured from the participant's first-person experience will be analyzed in relation to the neural data to infer how changes in patterns of neural activity over time relate to changes in experience. In addition, stimulation will be applied, at safe levels and timed according to "event boundaries" of the participant's experience, to determine whether memories of specific events can be enhanced. The proposed platform that allows for neural recording, direct brain stimulation, and synchronization with external, wearable devices will open an entirely new area of research at the intersection of computer science, engineering, cognition, and clinical neuroscience. These studies will launch and accelerate an emerging and pivotal area of research that will provide therapeutic interventions, proven in the real-world, for participants afflicted with debilitating cognitive disorders. This project will also make substantial contributions to education and outreach, including the development of K-12 classroom modules, interdisciplinary graduate training, outreach to industry partners in the neuromodulation field, and workshops at local Salt Lake City memory care communities.Development of this neural and first-person experience recording system will entail three collaborative research tasks: i) Synchronizing the Human Experience Relative to Neuronal Events: This module will develop a robust framework to record and synchronize neuronal activity along with internet of things (IoT) sensor data representing a broad subset of human sensory channels. The design will be portable such that the human experience can be reasoned about outside of a simulated lab environment. ii) Real-time Semantic Alignment between Human and IoT Perception: Reasoning about the complex relationships between neural biomarkers and the human experience captured by IoT sensing requires more than sensor synchronization. Neural-symbolic approaches that integrate the perception capabilities of deep learning with human logic will be leveraged to reason about the high-level complex spatiotemporal dependencies across a heterogeneous set of sensors. iii) Enhancing Episodic Memories of Real-world Experiences with DBS: Given a proper characterization of neural oscillations associated with event boundaries, the investigators will work to enhance episodic memories of real-world experiences with wireless deep brain stimulation (DBS) devices by directly stimulating the human brain. Under medical supervision, stimulation will be applied to the human amygdala at and between event boundaries in subjects with implanted stimulation devices as they encounter novel, 3D augmented reality objects while navigating a large-scale, real-world environment. Memory will be subsequently tested in laboratory and real-world settings. These three research areas will develop a situational understanding of neuronal activity in the context of human experience. They will further lay the foundation for future research directions in safe and effective stimulation of the brain in response to human experience in the wild.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.