HCC: Small: Modeling Human Communication Dynamics

面对面的交流是一个高度动态的过程，参与者相互交换和解释语言和手势信号。即使只有一个人说话，其他参与者也会通过手势、凝视、姿势和面部表情不断地在他们之间和说话者之间交换信息。为了正确解读高层次的交际信号，观察者需要将所有参与者的言语、细微的韵律变化和同步的手势综合起来。本研究旨在建立新一代的计算模型，以模拟社会互动中语言符号和非语言信号之间的相互依存关系。该计算框架具有广泛的适用性，包括人类社会行为的识别，机器人和虚拟人的自然动画的合成，改进的多媒体内容分析，以及社会和行为障碍的诊断（例如自闭症谱系障碍）。这项研究成果是一个重要的里程碑，补充了最近只关注两个组成部分的研究成果（例如，关注非语言信号和社会信号的社会信号处理）。提出的社会符号-信号的统一方法将为新的鲁棒和高效的计算感知算法铺平道路，这些算法能够识别高级交际行为（例如，意图和情感），并将为行为科学的研究人员提供新的计算工具。该研究将通过开发新的概率模型来共同捕获语言、手势和社会信号之间的相互依赖性，以及新颖的计算表示来推进这一努力，该模型集成了数据驱动处理和基于逻辑规则的方法（因此可以轻松地包括来自社会科学的先验知识）。将直接解决四个基本研究目标：符号-信号表示（语言和非语言的联合表示），建模社会相互依赖（多个参与者之间的交流信号的联合建模），信号解释的可变性（高水平交流信号的注释可变性），以及泛化和验证（不同交流信号和领域的泛化）。拟议的研究将使用户和具体的会话对话系统之间能够更自然地相互作用，影响计算机的使用方式，例如在辅导和文化和语言训练方面。这些软件和数据的潜在用途远远超出了本项目的范围，例如，可以对人类面对面（多模式）交流的社会方面或人类多模式处理的认知方面进行大规模的基于语料库的研究。根据研究人员过去分享研究软件开源、代码和语料库注释的经验，将向研究社区提供。这些共享的研究成果将对新的研究人员有价值，同时也是课程开发的重要教育材料。

Face-to-face communication is a highly dynamic process where participants mutually exchange and interpret linguistic and gestural signals. Even when only one person speaks at the time, other participants exchange information continuously amongst themselves and with the speaker through gesture, gaze, posture and facial expressions. To correctly interpret the high-level communicative signal, an observer needs to jointly integrate all spoken words, subtle prosodic changes and simultaneous gestures from all participants.The proposed effort endeavors to create a new generation of computational models for modeling the interdependence between linguistic symbols and nonverbal signals during social interactions. This computational framework has wide applicability, including the recognition of human social behaviors, the synthesis of natural animations for robots and virtual humans, improved multimedia content analysis, and the diagnosis of social and behavioral disorders (e.g., autism spectrum disorder). This research effort is an important milestone, complementary to recent research efforts focusing on only two components (e.g., social signal processing, which focuses on nonverbal and social signals). The proposed unified approach to Social-Symbols-Signals will pave the way for new robust and efficient computational perception algorithms able to recognize high-level communicative behaviors (e.g., intent and sentiments) and will enable new computational tools for researchers in behavioral sciences. The proposed research will advance this endeavor through the development of new probabilistic models for jointly capturing the interdependence between language, gestures and social signals, and novel computational representations, which integrates data-driven processing and logic rule-based approach (so that prior knowledge from social sciences can be easily included). Four fundamental research goals will be directly addressed: symbol-signal representation (joint representation of language and nonverbal), modeling social interdependence (joint modeling of communicative signals between multiple participants), variability in signal interpretations (variability with annotations of high-level communicative signals), and generalization and validation (generalization over different communicative signals and domains).The proposed research will enable more natural interaction between users and embodied conversational dialogue systems, impacting the way in which computers are used, for example, in tutoring and in cultural and language training. The potential uses of such software and data go far beyond the scope of this project, making it possible, for example, to perform large scale corpus-based studies about social aspects of human face-to-face (multimodal) communication, or cognitive aspects of human multimodal processing. Following the investigators' past experience with sharing research software open-source, code and corpus annotations will be made available to the research community. These shared research results will be valuable for new researchers as well as important educational material for course development.