CHS: Small: Collaborative Research: A Graph-Based Data Fusion Framework Towards Guiding A Hybrid Brain-Computer Interface

非侵入性脑机接口(BCI)的重大进展为某些残疾人提供了替代的通信手段，从而丰富了他们的生活。然而，当前的系统严重依赖于单峰技术，这限制了它们的性能和我们对集成神经动力学的理解，而集成神经动力学是正确解释多尺度神经功能所必需的。为了解决这一问题，已经提出使用混合(多模式)BCI，但迄今为止通过简单组合(例如，来自两个神经成像模式的特征集的串联)来利用多模式的互补优势的尝试仅产生了递增的进步；用于多模式信号的融合以高效且同时从多个感兴趣的信号中提取补充信息的可推广的计算数据驱动的方法仍然缺乏。这项研究将探索一种创新的方法，利用基于图形的数据融合框架，分别使用EEG和fNIR从电和血流动力学神经信号中获得互补的生理特征，从而实现混合非侵入性脑-机接口系统。项目成果将包括新的信号处理管道，并为主流用户应用的实用脑-机接口技术奠定基础。除了该项目的潜在社会影响外，该团队将专注于扩大对STEM的参与，并将吸引从K-12到研究生水平的学生。研究将涉及三个主要方面。将开发一种新的图论多模式数据融合框架，以系统地捕获混合模式的复杂拓扑特征和用户在同时调制感兴趣的电和血流动力学响应的双任务交互期间的意图。由于多通道技术在时空分辨率和信息内容方面都产生了内在的互补属性，因此该框架的目标是从隐藏在脑电信号和fNIRS信号中的复杂混合模式中捕获相应的互补协同拓扑特征，以用于高层提取用户意图。该框架将通过在现实世界环境中优化包含最高互信息的参数和渠道，在非交流的个人身上进行评估。最后，将提出一种概念上新的混合子空间过滤器，以最大化两类混合数据之间的距离并增强分类性能。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Major advances in non-invasive brain-computer interfaces (BCIs) have enriched the lives of persons with certain disabilities by providing them with alternative means of communication. However, current systems rely heavily on unimodal techniques that limit both their performance and our understanding of the integrated neural dynamics essential to properly explain multiscale neural functions. To address this issue it has been proposed to employ hybrid (multimodal) BCIs, but attempts to date to utilize the complementary benefits of multiple modalities through simple combinations (e.g., concatenation of feature sets from two neuroimaging modalities) have yielded only incremental advances; generalizable computational data-driven approaches for the fusion of multimodal signals to efficiently and simultaneously extract complementary information from multiple signals of interest remain lacking. This research will explore an innovative approach to a hybrid non-invasive BCI system that capitalizes on the complementary physiological features that can be obtained from electrical and hemodynamic neural signals using EEG and fNIRS respectively, with the help of a graph-based data fusion framework. Project outcomes will include novel signal processing pipelines and lay the foundation for practical BCI techniques for mainstream user applications. In addition to the project's potential societal impacts, the team will focus on broadening participation in STEM and will also engage students from K-12 through the graduate level.The research will involve three main thrusts. A novel graph theoretical multimodal data fusion framework will be developed to systematically capture complex topological features of hybrid patterns and user intentions during a dual-task interaction that concurrently modulates electrical and hemodynamic responses of interest. Because multimodal techniques create inherently complementary attributes in terms of both spatiotemporal resolution and information content, the framework will aim to capture the corresponding complementary synergistic topological features from the complex hybrid patterns hidden in EEG and fNIRS signals for the high-level abstraction of user intentions. The framework will be evaluated on non-communicative individuals by optimizing parameters and channels containing the highest mutual information, in real-world settings. Finally, a conceptually new hybrid subspace-based filter will be proposed to maximize the distance between two classes of hybrid data and enhance classification performance.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.