CHS: Small: Collaborative Research: A Graph-Based Data Fusion Framework Towards Guiding A Hybrid Brain-Computer Interface

非侵入性脑机接口（bci）的重大进展为某些残疾人提供了另一种交流方式，丰富了他们的生活。然而，目前的系统严重依赖于单峰技术，这限制了它们的性能和我们对综合神经动力学的理解，而这对于正确解释多尺度神经功能至关重要。为了解决这个问题，有人建议采用混合（多模式）脑机接口，但迄今为止，试图通过简单的组合（例如，从两种神经成像模式连接特征集）来利用多种模式的互补优势，只产生了渐进的进展；多模态信号融合的通用计算数据驱动方法仍然缺乏，无法同时有效地从多个感兴趣的信号中提取互补信息。本研究将探索一种混合无创脑机接口系统的创新方法，该系统利用EEG和fNIRS分别从电和血流动力学神经信号中获得的互补生理特征，并借助基于图的数据融合框架。项目成果将包括新的信号处理管道，并为主流用户应用的实际BCI技术奠定基础。除了该项目潜在的社会影响外，该团队还将专注于扩大STEM的参与，并将吸引从K-12到研究生阶段的学生。这项研究将涉及三个主要重点。将开发一种新的图理论多模态数据融合框架，以系统地捕获混合模式和用户意图的复杂拓扑特征，同时调节感兴趣的电和血流动力学响应。由于多模态技术在时空分辨率和信息内容方面创造了固有的互补属性，因此该框架旨在从隐藏在EEG和fNIRS信号中的复杂混合模式中捕获相应的互补协同拓扑特征，以便对用户意图进行高层抽象。该框架将在现实世界中通过优化包含最高相互信息的参数和渠道对非沟通个体进行评估。最后，提出了一种概念上的基于子空间的混合滤波器，以最大化两类混合数据之间的距离，提高分类性能。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Major advances in non-invasive brain-computer interfaces (BCIs) have enriched the lives of persons with certain disabilities by providing them with alternative means of communication. However, current systems rely heavily on unimodal techniques that limit both their performance and our understanding of the integrated neural dynamics essential to properly explain multiscale neural functions. To address this issue it has been proposed to employ hybrid (multimodal) BCIs, but attempts to date to utilize the complementary benefits of multiple modalities through simple combinations (e.g., concatenation of feature sets from two neuroimaging modalities) have yielded only incremental advances; generalizable computational data-driven approaches for the fusion of multimodal signals to efficiently and simultaneously extract complementary information from multiple signals of interest remain lacking. This research will explore an innovative approach to a hybrid non-invasive BCI system that capitalizes on the complementary physiological features that can be obtained from electrical and hemodynamic neural signals using EEG and fNIRS respectively, with the help of a graph-based data fusion framework. Project outcomes will include novel signal processing pipelines and lay the foundation for practical BCI techniques for mainstream user applications. In addition to the project's potential societal impacts, the team will focus on broadening participation in STEM and will also engage students from K-12 through the graduate level.The research will involve three main thrusts. A novel graph theoretical multimodal data fusion framework will be developed to systematically capture complex topological features of hybrid patterns and user intentions during a dual-task interaction that concurrently modulates electrical and hemodynamic responses of interest. Because multimodal techniques create inherently complementary attributes in terms of both spatiotemporal resolution and information content, the framework will aim to capture the corresponding complementary synergistic topological features from the complex hybrid patterns hidden in EEG and fNIRS signals for the high-level abstraction of user intentions. The framework will be evaluated on non-communicative individuals by optimizing parameters and channels containing the highest mutual information, in real-world settings. Finally, a conceptually new hybrid subspace-based filter will be proposed to maximize the distance between two classes of hybrid data and enhance classification performance.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.