HCC: Small: Modeling Human Communication Dynamics

HCC：小型：模拟人类沟通动态

• 批准号: 1118018
• 负责人: Louis-Philippe Morency
• 金额: $ 49.01万
• 依托单位: University of Southern California
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2011
• 资助国家: 美国
• 起止时间: 2011-08-15 至 2015-02-28
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1118018&HistoricalAwards=false
• 关键词: HCC; Small; Modeling; Human; Communication

Face-to-face communication is a highly dynamic process where participants mutually exchange and interpret linguistic and gestural signals. Even when only one person speaks at the time, other participants exchange information continuously amongst themselves and with the speaker through gesture, gaze, posture and facial expressions. To correctly interpret the high-level communicative signal, an observer needs to jointly integrate all spoken words, subtle prosodic changes and simultaneous gestures from all participants.The proposed effort endeavors to create a new generation of computational models for modeling the interdependence between linguistic symbols and nonverbal signals during social interactions. This computational framework has wide applicability, including the recognition of human social behaviors, the synthesis of natural animations for robots and virtual humans, improved multimedia content analysis, and the diagnosis of social and behavioral disorders (e.g., autism spectrum disorder). This research effort is an important milestone, complementary to recent research efforts focusing on only two components (e.g., social signal processing, which focuses on nonverbal and social signals). The proposed unified approach to Social-Symbols-Signals will pave the way for new robust and efficient computational perception algorithms able to recognize high-level communicative behaviors (e.g., intent and sentiments) and will enable new computational tools for researchers in behavioral sciences. The proposed research will advance this endeavor through the development of new probabilistic models for jointly capturing the interdependence between language, gestures and social signals, and novel computational representations, which integrates data-driven processing and logic rule-based approach (so that prior knowledge from social sciences can be easily included). Four fundamental research goals will be directly addressed: symbol-signal representation (joint representation of language and nonverbal), modeling social interdependence (joint modeling of communicative signals between multiple participants), variability in signal interpretations (variability with annotations of high-level communicative signals), and generalization and validation (generalization over different communicative signals and domains).The proposed research will enable more natural interaction between users and embodied conversational dialogue systems, impacting the way in which computers are used, for example, in tutoring and in cultural and language training. The potential uses of such software and data go far beyond the scope of this project, making it possible, for example, to perform large scale corpus-based studies about social aspects of human face-to-face (multimodal) communication, or cognitive aspects of human multimodal processing. Following the investigators' past experience with sharing research software open-source, code and corpus annotations will be made available to the research community. These shared research results will be valuable for new researchers as well as important educational material for course development.

面对面交流是一个高度动态的过程，参与者相互交换和解释语言和手势信号。即使当时只有一个人发言，其他参与者也会通过手势、凝视、姿势和面部表情在他们之间不断地交换信息，并与说话者交换信息。为了正确解释高级交际信号，观察者需要联合整合所有参与者的所有口语、细微的韵律变化和同时的手势。本研究旨在创建新一代计算模型，以模拟社会互动中语言符号和非语言信号之间的相互依赖。该计算框架具有广泛的适用性，包括人类社会行为的识别、机器人和虚拟人的自然动画合成、改进的多媒体内容分析以及社会和行为障碍(例如自闭症谱系障碍)的诊断。这项研究工作是一个重要的里程碑，补充了最近只关注两个组成部分的研究工作(例如，关注非语言和社会信号的社会信号处理)。所提出的社会符号-信号的统一方法将为能够识别高级交流行为(例如意图和情感)的新的健壮和高效的计算感知算法铺平道路，并将为行为科学的研究人员提供新的计算工具。拟议的研究将通过开发新的概率模型来共同捕捉语言、手势和社交信号之间的相互依赖，以及新的计算表示法来推动这一努力，该模型集成了数据驱动处理和基于逻辑规则的方法(以便可以轻松地纳入社会科学的先验知识)。四个基本的研究目标将直接涉及：符号-信号表示(语言和非语言的联合表示)，社会相互依赖的建模(多个参与者之间的交流信号的联合建模)，信号解释的可变性(与高级交流信号的注释的可变性)，以及概括和验证(对不同的交流信号和领域的概括)。拟议的研究将使用户和具体的对话系统之间能够进行更自然的交互，影响计算机的使用方式，例如在教学和文化和语言培训中。这类软件和数据的潜在用途远远超出了这个项目的范围，例如，有可能对人类面对面(多模式)交流的社会方面或人类多模式处理的认知方面进行大规模的基于语料库的研究。根据研究人员过去分享研究软件开源的经验，代码和语料库注释将向研究界提供。这些共享的研究成果将对新的研究人员以及课程开发的重要教育材料具有价值。