TherapAI - Analysis of nonverbal emotional expression in psychotherapy through artificial intelligence

背景：心理治疗中对情绪的关注与治疗结果和过程高度相关，特别是在抑郁症等情绪障碍中。除了患者或治疗师的个人情绪，他们的情绪互动(例如，情绪同步性；ES)起着重要作用。ES可以提供与治疗过程相关的信息，特别是对于情感知觉和表达障碍的患者，就像抑郁症一样。然而，对情绪和ES的研究主要是基于自我报告或小样本，因为手动评分很耗时。人工智能(AI)视频分析软件，如非语言行为分析器(NOVA)，可以自动、方便和改进评分过程，提供新的机会来检查情绪和心理治疗中的情绪交互。目的：本项目的目的是在抑郁症患者(F32，F33，被诊断为SCID-5)样本中，检查患者和治疗师的情绪和ES(由NOVA自动评估)与结果、辍学、会话到会话变化和过程变量(应对技能、治疗关系、情绪投入)的关系。同时，NOVA将根据研究过程中的实际应用结果进行调整，以适应临床应用的需要。特别是，功能的用户友好性和可理解性将得到改善。为了便于其他研究人员复制研究结果，应用模型将免费提供。方法：总样本包括N=200名治疗师和大约200名患者。1800个治疗视频(á50分钟)。Nova将接受培训，以检测患者和治疗师在视频中的情绪和情绪。为此，训练有素的人类评分员将在每个二人组的四个会话中手动编码情绪表达的价格和唤醒。基于这些手动评级，Nova将学习自动对视频中的情绪进行评级。模型将在70%(n=140)的数据中进行训练，并在其余30%(n=60)的数据中测试10次，以选择最佳模型。然后，Nova将使用最终的模型，在200个二元组的前十个疗程的所有可用视频中自动对患者和治疗师的情绪进行评级。之后，将计算ES。大约。将使用多水平线性模型和逻辑回归来测试情绪和ES的平均水平与结果、症状减轻、辍学、会话间变化以及过程变量(应对技能、治疗关系、情绪介入)之间的关系。讨论：由于情绪对心理障碍和治疗的重要性，本项目中开发的情绪自动分析扩展了心理治疗研究关于测量和预测治疗过程和结果的可能性。

Background: Focusing on emotions in psychotherapy is highly relevant for therapeutic outcome and process, especially in emotional disorders like depression. Beyond patient’s or therapist’s individual emotions, their emotional interaction (e.g., emotional synchrony; ES) plays an important role. ES can provide information relevant to the therapeutic process, especially for patients with disturbed perception and expression of emotions, as is the case with depression. However, studies of emotions and ES have mainly been based on self-reports or small samples, because manual ratings are time-consuming. Artificial intelligence (AI) video analysis software such as the Nonverbal Behavior Analyzer (NOVA) can automatize, facilitate, and improve the rating process, providing new opportunities to examine emotions and emotional interactions in psychotherapy.Aim: The aim of this project is to examine how patients’ and therapists’ emotions and ES, automatically assessed by NOVA, are related to outcome, dropout, session-to-session change, and process variables (coping skills, therapeutic relationship, emotional involvement) in a sample of patients with depression (F32, F33, diagnosed with the SCID-5). Simultaneously, NOVA will be adapted to the needs of clinical application based on the findings gained from practical application over the course of the study. In particular, user-friendliness and comprehensibility of the functions will be improved. To facilitate replicability of the study results by other researchers, the applied models will be made freely available.Method: The total sample consists of N = 200 therapists and patients with approx. 1,800 therapy videos (á 50 minutes). NOVA will be trained to detect patients’ and therapists’ emotions and ES in the videos. To this end, trained human raters will manually code valence and arousal of the emotional expression in four sessions per dyad. Based on these manual ratings, NOVA will learn to automatically rate emotions in videos. Models will be trained in 70% (n = 140) of the data and tested in the remaining 30% (n = 60) ten times to select the best model. Then, NOVA will use the final model to automatically rate patients’ and therapists’ emotions in all available videos of the first ten sessions of the 200 dyads. Afterwards, ES will be calculated. The approx. 1,800 sessions (10% missings expected) will be used to test how the average levels of emotions and ES are related to outcome, symptom reduction, dropout, session-to-session change, and process variables (coping skills, therapeutic relationship, emotional involvement) using multilevel linear models and logistic regressions.Discussion: Due to the importance of emotions for psychological disorders and therapies, an automated analysis of emotions developed in this project expands the possibilities of psychotherapy research regarding the measurement and prediction of treatment processes and outcome.