EAGER: EEG-based Cognitive-state Decoding for Interactive Virtual Reality

越来越多的经济实惠、高性能的虚拟现实（VR）耳机为教育、培训和治疗等应用创造了巨大的潜力。在许多应用中，能够感知用户的精神状态可以提供关键的好处。例如，虚拟现实环境可以使用脑电图（EEG）等大脑信号来推断用户的精神负荷或情绪状态；反过来，这可以用来改变训练任务的难度，使其更适合每个用户的独特体验。使用这种脑电图反馈不仅对训练有价值，而且在实际应用中提高人们的表现，包括航空、医疗、国防和驾驶。该项目的目标是开发将EEG传感器集成到当前VR头显中的方法和算法，为安装这些传感器提供一个合乎逻辑且不显眼的框架。然而，还有一些重要的挑战需要克服。例如，实验室中的脑电图传感器通常与导电凝胶一起使用，但对于VR头显来说，这些传感器需要在没有凝胶的“干燥”条件下可靠地工作。此外，在实验室环境中，运动不是问题，但处理脑电图数据的算法需要考虑人们使用耳机时头部和身体的运动。为了应对这些挑战，项目团队将以干式脑电图电极技术和运动伪影抑制算法的最新进展为基础，重点支持被动监测和认知状态反馈。这种被动反馈可能比主动脑电图反馈在虚拟环境中更有用，因为人们将使用其他方法直接与环境交互，因为被动脑电图感知比主动控制对较慢的响应时间和解码错误更健壮。先前的研究已经证明了脑电图在受控的实验室场景中用于认知状态解码的潜力，但在交互式VR环境中实现闭环神经反馈的脑电图集成需要解决三个关键的问题：(1)更实用和方便的脑电图干传感器能否达到与湿传感器相当的结果？(2)被动脑电认知状态解码能否对运动相关伪影具有鲁棒性？(3)这些解码方案是否可以推广到各种认知任务和闭环范式中？为了解决这些问题，经典的认知任务和更复杂的模拟器任务将作为新颖的交互式VR环境进行实施和测试。基于使用主动湿式脑电图传感器在交互式VR中成功表征运动工件和解码认知工作量的初步结果，本工作将进一步探索脑电图传感器与房间级VR头显的实际集成，以平衡数据质量、认知解码性能、设置和使用的便利性以及用户舒适度。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The increasing availability of affordable, high-performance virtual reality (VR) headsets creates great potential for applications including education, training, and therapy. In many applications, being able to sense a user's mental state could provide key benefits. For instance, VR environments could use brain signals such as the electroencephalogram (EEG) to infer aspects of the user's mental workload or emotional state; this, in turn, could be used to change the difficulty of a training task to make it better-suited to each user's unique experience. Using such EEG feedback could be valuable not just for training, but in improving people's performance in real applications including aviation, healthcare, defense, and driving. This project's goal is to develop methods and algorithms for integrating EEG sensors into current VR headsets, which provide a logical and unobtrusive framework for mounting these sensors. However, there are important challenges to overcome. For instance, EEG sensors in labs are typically used with a conducting gel, but for VR headsets these sensors will need to work reliably in "dry" conditions without the gel. Further, in lab settings, motion isn't an issue, but algorithms for processing the EEG data will need to account for people's head and body motion when they are using headsets. To address these challenges, the project team will build on recent advances in dry EEG electrode technologies and motion artifact suppression algorithms, focusing on supporting passive monitoring and cognitive state feedback. Such passive feedback is likely to be more usable in virtual environments than active EEG feedback, both because people will be using other methods to interact with the environment directly and because passive EEG sensing is more robust to slower response times and decoding errors than active control. Prior studies have demonstrated the potential of EEG for cognitive-state decoding in controlled laboratory scenarios, but practical EEG integration for closed-loop neurofeedback in interactive VR environments requires addressing three critical next questions: (1) can more-practical and convenient EEG dry sensors achieve comparable results to wet sensors?, (2) can passive EEG cognitive-state decoding be made robust to movement-related artifacts?, and (3) can these decoding schemes be generalized across a variety of cognitive tasks and to closed-loop paradigms? To address these questions, classical cognitive tasks and more-complex simulator tasks will be implemented and tested as novel, interactive VR environments. Building upon preliminary results that successfully characterized movement artifacts and decoded cognitive workload in interactive VR using active-wet EEG sensors, this work will further explore the practical integration of EEG sensors with room-scale VR headsets to balance data quality, cognitive decoding performance, ease of setup and use, and user comfort.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.