权益分类	功能权益	普通用户	{{item.name}}会员
{{category.name}}	{{benefitItem.name}}

CAREER: Learning Nonverbal Signatures

职业：学习非语言签名

基本信息

批准号：
1750439
负责人：
Louis-Philippe Morency
金额：
$ 55万
依托单位：
Carnegie-Mellon University
依托单位国家：
美国
项目类别：
Continuing Grant
财政年份：
2018
资助国家：
美国
起止时间：
2018-02-15 至 2023-01-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1750439&HistoricalAwards=false
关键词：
CAREER Learning Nonverbal Signatures

项目摘要

Nonverbal communication is an essential component of our daily life. A quick eyebrow frown from a student will inform the teacher of possible confusion. A tilt of the head may show doubt during negotiation. A patient's smile dynamics can even predict their future recovery. This project will build innovative technologies to allow computers to understand subtle nonverbal behaviors of human users. The main novelty of this project will be in its capacity to learn the inherent variability between individuals on how nonverbal behaviors are expressed. The same emotion can be expressed very differently by different people. A nonverbal signature is how a specific person gestures visually when talking with other people. By recognizing these nonverbal communicative behaviors such as facial expressions, head gestures and eye contact, computers and mobile devices will be able to better understand the user's emotions and affective states. In mental health treatment, the developed technologies have the potential to change how doctors assess disorders with new behavioral objective measures and possibily to predict eventual relapse for the patient. For researchers and animators of virtual characters (e.g., movie studies), the nonverbal signatures will enable an "a la carte" selection of the virtual human nonverbal signature that best aligns with the character's profile and personality. In education and online learning, the student's nonverbal signature will help to assess engagement and perceive affective states related to success in learning. This project will advocate a novel paradigm for learning visual representations of human nonverbal behaviors: a major focus on intra-personal variability followed by analysis of group structures and eventual learning of visual representations generalizable across the population. The research methodology will be motivated by well-studied concepts of idiosyncrasy in nonverbal behavior expressions. The project will introduce the concept of nonverbal signatures which are low-dimensional computational representations of an individual's nonverbal behaviors contextualized by the verbal content and affective context. This project will address four fundamental research challenges: (1) personalized nonverbal embedding -- learning computational representations that summarize the individual variations in nonverbal appearance, (2) learning nonverbal signatures -- contextualizing the nonverbal behaviors with verbal and affective cues to learn more effective representations, (3) signature portfolio analysis -- discovering structure, prototypes and idiosyncrasy from a collection of nonverbal signatures, and (4) generalized nonverbal representations -- learning generalizable nonverbal representations able to adapt to new individuals. These four research aims will be complemented by a comprehensive evaluation plan to include four intermediate evaluations and a continuous overarching evaluation. This research effort will open the door to new sources of human-centric data where accurate interpretation of nonverbal behaviors is essential.

非语言交流是我们日常生活中不可或缺的组成部分。一个学生快速的皱眉会告诉老师可能的困惑。在谈判中歪着头可能表示怀疑。患者的微笑动态甚至可以预测他们未来的康复情况。该项目将建立创新技术，使计算机能够理解人类用户微妙的非语言行为。这个项目的主要新奇之处在于它能够学习个体之间关于非语言行为如何表达的固有差异。同样的情绪，不同的人表达出来的方式也会大不相同。非语言签名是指特定的人在与其他人交谈时如何做出视觉手势。通过识别这些非语言交流行为，如面部表情、头部姿势和眼神交流，计算机和移动的设备将能够更好地理解用户的情绪和情感状态。在心理健康治疗中，已开发的技术有可能改变医生使用新的行为客观指标评估疾病的方式，并有可能预测患者最终的复发。对于虚拟角色的研究人员和动画师（例如，电影研究），非语言签名将使得能够“点菜”选择与角色的简档和个性最一致的虚拟人非语言签名。在教育和在线学习中，学生的非语言签名将有助于评估参与度并感知与学习成功相关的情感状态。该项目将倡导一种学习人类非语言行为的视觉表征的新范式：主要关注个人内部的差异，然后分析群体结构，并最终学习可在人群中推广的视觉表征。研究方法的动机是充分研究的非语言行为表达的特质概念。该项目将引入非语言签名的概念，这是一个人的非语言行为的低维计算表示的语言内容和情感背景。该项目将解决四个基础研究挑战：（1）个性化的非语言嵌入-学习总结非语言外观中的个体变化的计算表示，（2）学习非语言签名-将非语言行为与语言和情感线索联系起来以学习更有效的表示，（3）签名组合分析-发现结构，从一系列非语言特征中获得原型和特质，以及（4）广义非语言表征-学习能够适应新个体的可推广的非语言表征。这四个研究目标将辅之以一个全面评价计划，其中包括四个中期评价和一个持续的总体评价。这项研究工作将为以人为中心的数据的新来源打开大门，在这些数据中，对非语言行为的准确解释至关重要。

项目成果

期刊论文数量（38）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

MOSEAS: A Multimodal Language Dataset for Spanish, Portuguese, German and French.

MOSEAS：西班牙语、葡萄牙语、德语和法语的多模态语言数据集。

DOI：
发表时间：
2020
期刊：
2020
影响因子：
0
作者：
Zadeh, A.;Cao, Y.;Hessner, S.;Liang, P.;Poria, S.;Morency, L.-P.
通讯作者：
Morency, L.-P.

Speaker-Follower Models for Vision-and-Language Navigation

DOI：
发表时间：
2018-06
期刊：
ArXiv
影响因子：
0
作者：
Daniel Fried;Ronghang Hu;Volkan Cirik;Anna Rohrbach;Jacob Andreas;Louis-Philippe Morency;Taylor Berg-Kirkpatrick;Kate Saenko;D. Klein;Trevor Darrell
通讯作者：
Daniel Fried;Ronghang Hu;Volkan Cirik;Anna Rohrbach;Jacob Andreas;Louis-Philippe Morency;Taylor Berg-Kirkpatrick;Kate Saenko;D. Klein;Trevor Darrell

Beyond Additive Fusion: Learning Non-Additive Multimodal Interactions

超越加法融合：学习非加法多模态交互

DOI：
发表时间：
2022
期刊：
Findings of the Association for Computational Linguistics: EMNLP 2022
影响因子：
0
作者：
Wörtwein, Torsten;Sheeber, Lisa;Allen, Nicholas;Cohn, Jeffrey;Morency, Louis-Philippe
通讯作者：
Morency, Louis-Philippe

Strong and Simple Baselines for Multimodal Utterance Embeddings

DOI：
10.18653/v1/n19-1267
发表时间：
2019-05
期刊：
影响因子：
0
作者：
P. Liang;Y. Lim;Yao-Hung Hubert Tsai;R. Salakhutdinov;Louis-Philippe Morency
通讯作者：
P. Liang;Y. Lim;Yao-Hung Hubert Tsai;R. Salakhutdinov;Louis-Philippe Morency

Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence