TherapAI - Analysis of nonverbal emotional expression in psychotherapy through artificial intelligence

背景：心理治疗中对情绪的关注与治疗结果和治疗过程密切相关，尤其是在抑郁症等情绪障碍中。除了患者或治疗师的个人情绪外，他们的情绪互动（例如，情绪同步；ES）也起着重要作用。ES可以提供与治疗过程相关的信息，特别是对于情绪感知和表达紊乱的患者，就像抑郁症的情况一样。然而，情绪和社会行为的研究主要是基于自我报告或小样本，因为人工评分很耗时。非语言行为分析器（NOVA）等人工智能（AI）视频分析软件可以自动化、促进和改进评分过程，为心理治疗中检查情绪和情绪互动提供了新的机会。目的：本项目的目的是研究由NOVA自动评估的患者和治疗师的情绪和ES如何与抑郁症患者样本（F32, F33，诊断为SCID-5）的结果，辍学，会话到会话的变化和过程变量（应对技能，治疗关系，情感参与）相关。同时，NOVA将根据在研究过程中从实际应用中获得的发现来适应临床应用的需要。特别是，将改进功能的用户友好性和可理解性。为了便于其他研究人员复制研究结果，应用的模型将免费提供。方法：总样本由N = 200名治疗师和患者组成。1800个治疗视频（<s:1> 50分钟）。NOVA将接受训练，以检测患者和治疗师在视频中的情绪和情绪。为此，训练有素的人类评价员将在每对偶四次中手动编码情绪表达的效价和唤醒。基于这些手动评级，NOVA将学会自动对视频中的情绪进行评级。对70% （n = 140）的数据进行模型训练，对剩余30% （n = 60）的数据进行10次测试，选出最佳模型。然后，NOVA将使用最终的模型对200对夫妇的前10次会议的所有可用视频中的患者和治疗师的情绪进行自动评分。然后计算ES。约。将使用1800个疗程（预计缺失10%）来测试情绪和ES的平均水平如何与结果、症状减轻、辍学、疗程到疗程的变化和过程变量（应对技能、治疗关系、情绪参与）相关，使用多层次线性模型和逻辑回归。讨论：由于情绪对心理障碍和治疗的重要性，本项目开发的情绪自动分析扩展了心理治疗研究在治疗过程和结果的测量和预测方面的可能性。

Background: Focusing on emotions in psychotherapy is highly relevant for therapeutic outcome and process, especially in emotional disorders like depression. Beyond patient’s or therapist’s individual emotions, their emotional interaction (e.g., emotional synchrony; ES) plays an important role. ES can provide information relevant to the therapeutic process, especially for patients with disturbed perception and expression of emotions, as is the case with depression. However, studies of emotions and ES have mainly been based on self-reports or small samples, because manual ratings are time-consuming. Artificial intelligence (AI) video analysis software such as the Nonverbal Behavior Analyzer (NOVA) can automatize, facilitate, and improve the rating process, providing new opportunities to examine emotions and emotional interactions in psychotherapy.Aim: The aim of this project is to examine how patients’ and therapists’ emotions and ES, automatically assessed by NOVA, are related to outcome, dropout, session-to-session change, and process variables (coping skills, therapeutic relationship, emotional involvement) in a sample of patients with depression (F32, F33, diagnosed with the SCID-5). Simultaneously, NOVA will be adapted to the needs of clinical application based on the findings gained from practical application over the course of the study. In particular, user-friendliness and comprehensibility of the functions will be improved. To facilitate replicability of the study results by other researchers, the applied models will be made freely available.Method: The total sample consists of N = 200 therapists and patients with approx. 1,800 therapy videos (á 50 minutes). NOVA will be trained to detect patients’ and therapists’ emotions and ES in the videos. To this end, trained human raters will manually code valence and arousal of the emotional expression in four sessions per dyad. Based on these manual ratings, NOVA will learn to automatically rate emotions in videos. Models will be trained in 70% (n = 140) of the data and tested in the remaining 30% (n = 60) ten times to select the best model. Then, NOVA will use the final model to automatically rate patients’ and therapists’ emotions in all available videos of the first ten sessions of the 200 dyads. Afterwards, ES will be calculated. The approx. 1,800 sessions (10% missings expected) will be used to test how the average levels of emotions and ES are related to outcome, symptom reduction, dropout, session-to-session change, and process variables (coping skills, therapeutic relationship, emotional involvement) using multilevel linear models and logistic regressions.Discussion: Due to the importance of emotions for psychological disorders and therapies, an automated analysis of emotions developed in this project expands the possibilities of psychotherapy research regarding the measurement and prediction of treatment processes and outcome.