EAGER: EEG-based Cognitive-state Decoding for Interactive Virtual Reality

经济实惠的高性能虚拟现实（VR）头显的日益普及为教育、培训和治疗等应用创造了巨大潜力。在许多应用中，能够感知用户的精神状态可以提供关键的好处。例如，VR环境可以使用脑电信号（如脑电图）来推断用户的心理工作量或情绪状态;这反过来又可以用来改变训练任务的难度，使其更适合每个用户的独特体验。 使用这种EEG反馈不仅对训练有价值，而且对提高人们在真实的应用中的表现有价值，包括航空，医疗保健，国防和驾驶。该项目的目标是开发将EEG传感器集成到当前VR头显中的方法和算法，为安装这些传感器提供一个逻辑和不显眼的框架。然而，还有一些重要的挑战需要克服。例如，实验室中的EEG传感器通常与导电凝胶一起使用，但对于VR头显，这些传感器需要在没有凝胶的“干燥”条件下可靠地工作。此外，在实验室环境中，运动不是问题，但处理EEG数据的算法需要考虑人们在使用耳机时的头部和身体运动。为了应对这些挑战，项目团队将利用干EEG电极技术和运动伪影抑制算法的最新进展，重点支持被动监测和认知状态反馈。这种被动反馈可能比主动EEG反馈在虚拟环境中更有用，这既是因为人们将使用其他方法直接与环境交互，也是因为被动EEG感测比主动控制对较慢的响应时间和解码错误更具鲁棒性。先前的研究已经证明了EEG在受控实验室场景中用于认知状态解码的潜力，但是在交互式VR环境中用于闭环神经反馈的实际EEG集成需要解决三个关键的下一个问题：（1）更实用和方便的EEG干传感器能否实现与湿传感器相当的结果？(2)被动EEG认知状态解码是否可以对运动相关伪影具有鲁棒性？以及（3）这些解码方案是否可以推广到各种认知任务和闭环范式？ 为了解决这些问题，经典的认知任务和更复杂的模拟器任务将作为新颖的交互式VR环境进行实施和测试。在使用主动湿EEG传感器成功表征交互式VR中的运动伪影和解码认知工作量的初步结果的基础上，这项工作将进一步探索EEG传感器与房间级VR耳机的实际集成，以平衡数据质量，认知解码性能，易于设置和使用，该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The increasing availability of affordable, high-performance virtual reality (VR) headsets creates great potential for applications including education, training, and therapy. In many applications, being able to sense a user's mental state could provide key benefits. For instance, VR environments could use brain signals such as the electroencephalogram (EEG) to infer aspects of the user's mental workload or emotional state; this, in turn, could be used to change the difficulty of a training task to make it better-suited to each user's unique experience. Using such EEG feedback could be valuable not just for training, but in improving people's performance in real applications including aviation, healthcare, defense, and driving. This project's goal is to develop methods and algorithms for integrating EEG sensors into current VR headsets, which provide a logical and unobtrusive framework for mounting these sensors. However, there are important challenges to overcome. For instance, EEG sensors in labs are typically used with a conducting gel, but for VR headsets these sensors will need to work reliably in "dry" conditions without the gel. Further, in lab settings, motion isn't an issue, but algorithms for processing the EEG data will need to account for people's head and body motion when they are using headsets. To address these challenges, the project team will build on recent advances in dry EEG electrode technologies and motion artifact suppression algorithms, focusing on supporting passive monitoring and cognitive state feedback. Such passive feedback is likely to be more usable in virtual environments than active EEG feedback, both because people will be using other methods to interact with the environment directly and because passive EEG sensing is more robust to slower response times and decoding errors than active control. Prior studies have demonstrated the potential of EEG for cognitive-state decoding in controlled laboratory scenarios, but practical EEG integration for closed-loop neurofeedback in interactive VR environments requires addressing three critical next questions: (1) can more-practical and convenient EEG dry sensors achieve comparable results to wet sensors?, (2) can passive EEG cognitive-state decoding be made robust to movement-related artifacts?, and (3) can these decoding schemes be generalized across a variety of cognitive tasks and to closed-loop paradigms? To address these questions, classical cognitive tasks and more-complex simulator tasks will be implemented and tested as novel, interactive VR environments. Building upon preliminary results that successfully characterized movement artifacts and decoded cognitive workload in interactive VR using active-wet EEG sensors, this work will further explore the practical integration of EEG sensors with room-scale VR headsets to balance data quality, cognitive decoding performance, ease of setup and use, and user comfort.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.