MultiSense—Context-Aware Nonverbal Behavior Analysis Framework: A Psychological Distress Use Case
MultiSense—Context-Aware Nonverbal Behavior Analysis Framework: A Psychological Distress Use Case
复制标题
MultiSense - 上下文感知非语言行为分析框架：心理困扰用例
DOI：
--
复制
发表时间：
2017
影响因子：
11.2
通讯作者：
Louis
中科院分区：
文献类型：
--
作者：
Giota Stratou;Louis
在面对面的互动中，人们自然而然地将面部表情和身体姿势等非语言行为作为对话的一部分，以推断对话者的交际意图或情绪状态。对这些非语言行为的解释通常会受到互动线索的影响，如先前的口头问题、一般讨论话题或物理环境。创造能够理解或参与这种面对面社交互动的计算机的关键一步是开发一个计算平台，将非语言行为同步识别为互动语境的一部分。在这一平台中，声音和视觉通道的信息应该仔细同步和快速处理。同时，应该记住并整合上下文和交互线索，以便更好地解释非语言(和语言)行为。在本文中，我们介绍了一个实时计算框架，MultiSense，它为基于上下文的非言语行为分析提供了灵活而高效的同步方法。多重意义被设计成利用来自双方对话者(例如，来自计算机和人类参与者)的交互线索，并在解释非言语行为时整合该语境信息。多感知还可以吸收完全交互中的行为，并总结用户观察到的情感状态。我们通过一个来自精神健康领域的具体用例展示了新框架的能力，在该领域，多感知被用作评估诸如抑郁和创伤后应激障碍(PTSD)等心理痛苦指标的决策支持工具的一部分。在这种情况下，MultiSense不仅从非语言行为中推断出心理痛苦指标，而且还将用户状态实时广播给虚拟代理(即数字采访者)，该虚拟代理旨在对人类参与者进行半结构化采访。我们的实验表明了我们的多通道同步方法的附加值，也证明了在推断遇险指标时多意义语境解释的重要性。
During face-to-face interactions, people naturally integrate nonverbal behaviors such as facial expressions and body postures as part of the conversation to infer the communicative intent or emotional state of their interlocutor. The interpretation of these nonverbal behaviors will often be contextualized by interactional cues such as the previous spoken question, the general discussion topic or the physical environment. A critical step in creating computers able to understand or participate in this type of social face-to-face interactions is to develop a computational platform to synchronously recognize nonverbal behaviors as part of the interactional context. In this platform, information for the acoustic and visual modalities should be carefully synchronized and rapidly processed. At the same time, contextual and interactional cues should be remembered and integrated to better interpret nonverbal (and verbal) behaviors. In this article, we introduce a real-time computational framework, MultiSense, which offers flexible and efficient synchronization approaches for context-based nonverbal behavior analysis. MultiSense is designed to utilize interactional cues from both interlocutors (e.g., from the computer and the human participant) and integrate this contextual information when interpreting nonverbal behaviors. MultiSense can also assimilate behaviors over a full interaction and summarize the observed affective states of the user. We demonstrate the capabilities of the new framework with a concrete use case from the mental health domain where MultiSense is used as part of a decision support tool to assess indicators of psychological distress such as depression and post-traumatic stress disorder (PTSD). In this scenario, MultiSense not only infers psychological distress indicators from nonverbal behaviors but also broadcasts the user state in real-time to a virtual agent (i.e., a digital interviewer) designed to conduct semi-structured interviews with human participants. Our experiments show the added value of our multimodal synchronization approaches and also demonstrate the importance of MultiSense contextual interpretation when inferring distress indicators.
影响因子：
4.6
作者：
Reed, Lawrence Ian;Sayette, Michael A.;Cohn, Jeffrey F.
通讯作者：
Cohn, Jeffrey F.
影响因子：
5.9
作者：
Girard JM;Cohn JF
通讯作者：
Cohn JF
影响因子：
4.6
作者：
Rottenberg, J;Gross, JJ;Gotlib, IH
通讯作者：
Gotlib, IH