Extracting Multimodal Embeddings via Supervised Contrastive Learning for Psychological Screening
Extracting Multimodal Embeddings via Supervised Contrastive Learning for Psychological Screening
复制标题
DOI：
10.1109/acii55700.2022.9953836
复制
发表时间：
2022-10
期刊：
影响因子：
--
通讯作者：
Manasa Kalanadhabhatta;Adrelys Mateo Santana;Deepa Ganesan;Tauhidur Rahman;Adam S. Grabell
中科院分区：
文献类型：
--
作者：
Manasa Kalanadhabhatta;Adrelys Mateo Santana;Deepa Ganesan;Tauhidur Rahman;Adam S. Grabell
儿童早期心理障碍的诊断是至关重要的，因为它们严重影响儿童的学业和社会技能以及一般的适应功能。可穿戴和基于视频的系统有可能以神经生理和行为信号的形式收集重要的诊断信息。然而，要从多模态数据流中准确预测心理障碍状态，就需要将它们组合成有意义的特征进行分类模型。在这项工作中，我们提出了一种多任务监督对比学习方法，从功能性近红外光谱、皮肤电反应和在沮丧诱导任务中收集的面部视频数据中学习有用的多模态嵌入。生成的嵌入能够准确地推断情绪调节相关的心理障碍，F1得分为0.91，对幼儿心理健康诊断具有重要意义。
The diagnosis of psychological disorders in early childhood is of utmost importance given their severe impact on children's academic and social skills as well as general adaptive functioning. Wearable and video-based systems have the potential to collect important diagnostic information in the form of neurophysiological and behavioral signals. However, accurate prediction of psychological disorder status from multimodal data streams necessitates their combination into meaningful features for classification models. In this work, we present a multitask supervised contrastive learning approach to learn useful multimodal embeddings from functional Near-Infrared Spectroscopy, galvanic skin response, and facial video data collected during a frustration-inducing task. The generated embeddings are able to accurately infer emotion regulation-related psychological disorders with an F1 score of 0.91, having significant implications for early-childhood mental health diagnoses.