CHS: Small: Collaborative Research: A Graph-Based Data Fusion Framework Towards Guiding A Hybrid Brain-Computer Interface

非侵入性脑机接口的重大进展为某些残疾人提供了替代性通信手段，丰富了他们的生活。 然而，目前的系统严重依赖于单峰技术，限制了它们的性能和我们对正确解释多尺度神经功能所必需的集成神经动力学的理解。 为了解决这个问题，已经提出采用混合（多模式）BCI，但是迄今为止尝试通过简单组合（例如，来自两种神经成像模态的特征集的级联）仅产生了渐进的进步;用于多模态信号的融合以有效地且同时地从多个感兴趣的信号提取互补信息的可推广的计算数据驱动的方法仍然缺乏。 这项研究将探索一种创新的方法，以混合非侵入性BCI系统，利用互补的生理功能，可以从电和血液动力学神经信号，分别使用EEG和fNIRS，与基于图形的数据融合框架的帮助。项目成果将包括新颖的信号处理管道，并为主流用户应用的实用BCI技术奠定基础。 除了该项目的潜在社会影响外，该团队还将专注于扩大STEM的参与，并将吸引从K-12到研究生水平的学生。该研究将涉及三个主要方面。 将开发一种新的图论多模态数据融合框架，以系统地捕获复杂的拓扑特征的混合模式和用户的意图，在双任务的相互作用，同时调制感兴趣的电和血液动力学反应。 由于多模态技术创造内在的互补属性的时空分辨率和信息内容，该框架的目标是捕捉相应的互补协同拓扑功能，从复杂的混合模式隐藏在EEG和fNIRS信号的高层次抽象的用户意图。 该框架将通过优化参数和包含最高互信息的通道，在现实世界中的设置，对非沟通的个人进行评估。 最后，一个概念上的新的混合子空间为基础的过滤器将提出最大限度地提高两类混合数据之间的距离，提高classificationperformance.This奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

Major advances in non-invasive brain-computer interfaces (BCIs) have enriched the lives of persons with certain disabilities by providing them with alternative means of communication. However, current systems rely heavily on unimodal techniques that limit both their performance and our understanding of the integrated neural dynamics essential to properly explain multiscale neural functions. To address this issue it has been proposed to employ hybrid (multimodal) BCIs, but attempts to date to utilize the complementary benefits of multiple modalities through simple combinations (e.g., concatenation of feature sets from two neuroimaging modalities) have yielded only incremental advances; generalizable computational data-driven approaches for the fusion of multimodal signals to efficiently and simultaneously extract complementary information from multiple signals of interest remain lacking. This research will explore an innovative approach to a hybrid non-invasive BCI system that capitalizes on the complementary physiological features that can be obtained from electrical and hemodynamic neural signals using EEG and fNIRS respectively, with the help of a graph-based data fusion framework. Project outcomes will include novel signal processing pipelines and lay the foundation for practical BCI techniques for mainstream user applications. In addition to the project's potential societal impacts, the team will focus on broadening participation in STEM and will also engage students from K-12 through the graduate level.The research will involve three main thrusts. A novel graph theoretical multimodal data fusion framework will be developed to systematically capture complex topological features of hybrid patterns and user intentions during a dual-task interaction that concurrently modulates electrical and hemodynamic responses of interest. Because multimodal techniques create inherently complementary attributes in terms of both spatiotemporal resolution and information content, the framework will aim to capture the corresponding complementary synergistic topological features from the complex hybrid patterns hidden in EEG and fNIRS signals for the high-level abstraction of user intentions. The framework will be evaluated on non-communicative individuals by optimizing parameters and channels containing the highest mutual information, in real-world settings. Finally, a conceptually new hybrid subspace-based filter will be proposed to maximize the distance between two classes of hybrid data and enhance classification performance.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.