HCC: Small: Modeling Human Communication Dynamics

面对面交流是一个高度动态的过程，参与者相互交换和解释语言和手势信号。即使当时只有一个人说话，其他参与者也会通过手势、凝视、姿势和面部表情在他们之间以及与说话者之间不断交换信息。为了正确地解释高层次的交际信号，观察者需要联合整合所有的口语单词，微妙的韵律变化和同步的手势从所有participator.The拟议的努力，努力创建一个新一代的计算模型建模语言符号和非语言信号之间的相互依存关系在社会交往。这种计算框架具有广泛的适用性，包括人类社会行为的识别，机器人和虚拟人的自然动画的合成，改进的多媒体内容分析，以及社会和行为障碍的诊断（例如，自闭症谱系障碍）。这项研究工作是一个重要的里程碑，补充了最近只关注两个组成部分的研究工作（例如，社会信号处理，专注于非语言和社会信号）。所提出的社会符号信号的统一方法将为新的鲁棒和高效的计算感知算法铺平道路，该算法能够识别高级通信行为（例如，意图和情感），并将为行为科学研究人员提供新的计算工具。拟议的研究将通过开发新的概率模型来推进这一奋进，以共同捕捉语言，手势和社会信号之间的相互依赖关系，以及新的计算表示，它集成了数据驱动的处理和基于逻辑规则的方法（以便可以轻松地包括社会科学的先验知识）。四个基本的研究目标将直接解决：符号信号表示法（语言和非语言的联合表征），建模社会相互依存（多个参与者之间的通信信号的联合建模），信号解释的可变性（具有高级通信信号的注释的可变性），推广和验证（对不同通信信号和域的概括）拟议的研究将使用户和嵌入式会话对话系统之间的互动更加自然，影响计算机的使用方式，例如，在辅导和文化和语言培训中。这些软件和数据的潜在用途远远超出了本项目的范围，例如，可以对人类面对面（多模态）交流的社会方面或人类多模态处理的认知方面进行大规模基于语料库的研究。根据研究人员过去共享研究软件开源的经验，代码和语料库注释将提供给研究社区。这些共享的研究成果将对新的研究人员以及课程开发的重要教育材料有价值。

Face-to-face communication is a highly dynamic process where participants mutually exchange and interpret linguistic and gestural signals. Even when only one person speaks at the time, other participants exchange information continuously amongst themselves and with the speaker through gesture, gaze, posture and facial expressions. To correctly interpret the high-level communicative signal, an observer needs to jointly integrate all spoken words, subtle prosodic changes and simultaneous gestures from all participants.The proposed effort endeavors to create a new generation of computational models for modeling the interdependence between linguistic symbols and nonverbal signals during social interactions. This computational framework has wide applicability, including the recognition of human social behaviors, the synthesis of natural animations for robots and virtual humans, improved multimedia content analysis, and the diagnosis of social and behavioral disorders (e.g., autism spectrum disorder). This research effort is an important milestone, complementary to recent research efforts focusing on only two components (e.g., social signal processing, which focuses on nonverbal and social signals). The proposed unified approach to Social-Symbols-Signals will pave the way for new robust and efficient computational perception algorithms able to recognize high-level communicative behaviors (e.g., intent and sentiments) and will enable new computational tools for researchers in behavioral sciences. The proposed research will advance this endeavor through the development of new probabilistic models for jointly capturing the interdependence between language, gestures and social signals, and novel computational representations, which integrates data-driven processing and logic rule-based approach (so that prior knowledge from social sciences can be easily included). Four fundamental research goals will be directly addressed: symbol-signal representation (joint representation of language and nonverbal), modeling social interdependence (joint modeling of communicative signals between multiple participants), variability in signal interpretations (variability with annotations of high-level communicative signals), and generalization and validation (generalization over different communicative signals and domains).The proposed research will enable more natural interaction between users and embodied conversational dialogue systems, impacting the way in which computers are used, for example, in tutoring and in cultural and language training. The potential uses of such software and data go far beyond the scope of this project, making it possible, for example, to perform large scale corpus-based studies about social aspects of human face-to-face (multimodal) communication, or cognitive aspects of human multimodal processing. Following the investigators' past experience with sharing research software open-source, code and corpus annotations will be made available to the research community. These shared research results will be valuable for new researchers as well as important educational material for course development.