Modality Fusion Network and Personalized Attention in Momentary Stress Detection in the Wild
Modality Fusion Network and Personalized Attention in Momentary Stress Detection in the Wild
复制标题
DOI：
10.1109/acii52823.2021.9597459
复制
发表时间：
2021-07
期刊：
影响因子：
--
通讯作者：
Han Yu;T. Vaessen;I. Myin‐Germeys;Akane Sano
中科院分区：
文献类型：
--
作者：
Han Yu;T. Vaessen;I. Myin‐Germeys;Akane Sano
日常生活中的多模态可穿戴生理数据已被用于估计自我报告的压力标签。然而，数据收集中缺少数据模式，使得利用所有收集的样本具有挑战性。此外，个体之间的异构传感器数据和标签增加了构建鲁棒压力检测模型的挑战。在本文中，我们提出了一个模态融合网络（MFN）训练模型和推断自我报告的二进制压力标签在完整和不完整的模态条件。此外，我们应用了个性化的注意力（PA）策略，以利用个性化的表示沿着与广义的一刀切的模型。我们在多模态可穿戴传感器数据集（N=41）上评估了我们的方法，包括皮肤电反应（GSR）和心电图（ECG）。与使用具有完整模态的样本的基线方法相比，MFN的性能在f1分数上提高了1.6%。另一方面，与之前最先进的迁移学习策略（29.3 MB）相比，所提出的PA策略显示出高2.3%的压力检测f1分数和高达70%的个性化模型参数大小（9.1 MB）减少。我们提出的模型结构和实现的细节在https://github.com/comp-well-org/Modality-Fusion-Network-with-Personalized-Attention上共享。
Multimodal wearable physiological data in daily life have been used to estimate self-reported stress labels. However, missing data modalities in data collection makes it challenging to leverage all the collected samples. Besides, heterogeneous sensor data and labels among individuals add challenges in building robust stress detection models. In this paper, we proposed a modality fusion network (MFN) to train models and infer self-reported binary stress labels under both complete and incomplete modality condition. In addition, we applied a personalized attention (PA) strategy to leverage personalized representation along with the generalized one-size-fits-all model. We evaluated our methods on a multimodal wearable sensor dataset (N=41) including galvanic skin response (GSR) and electrocardiogram (ECG). Compared to the baseline method using the samples with complete modalities, the performance of the MFN improved by 1.6% in f1-scores. On the other hand, the proposed PA strategy showed a 2.3% higher stress detection f1-score and approximately up to 70% reduction in personalized model parameter size (9.1 MB) compared to the previous state-of-the-art transfer learning strategy (29.3 MB). The details of our proposed model structure and implementation are shared at https://github.com/comp-well-org/Modality-Fusion-Network-with-Personalized-Attention.