CAREER: Advanced Knowledge Extraction of Affective Behaviors During Natural Human Interaction

职业：人类自然互动过程中情感行为的高级知识提取

• 批准号: 1453781
• 负责人: Carlos Busso
• 金额: $ 49.59万
• 依托单位: University of Texas at Dallas
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-09-01 至 2021-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1453781&HistoricalAwards=false
• 关键词: CAREER; Advanced; Knowledge; Extraction; Affective

Identifying and characterizing emotional behaviors are challenging but very important research topics for enriched speech-derived analytics and human-computer interaction. This CAREER project aims to create novel algorithms to recognize spontaneous affective behaviors from speech that capture the underlying externalization process of emotions and generalize to recordings of human interactions collected under real-world conditions. The lack of generalization of current speech emotion algorithms to recognize expressive behaviors during natural human interaction is the key barrier to deploying affective-aware technology in real-life applications. Under a theoretical framework grounded in the nonuniform externalization of expressive behaviors, the project brings transformative solutions to address this problem. The proposed models and algorithms promise insights to explore and extend theories in linguistic
 and paralinguistic human behaviors. Several new scientific avenues can emerge that serve as truly innovative advancements that will impact applications in security and defense, next generation of advanced user interfaces, health behavior informatics, and education. The role of human centered technologies, especially contextualized in applications of direct societal relevance, can inspire young 
scholars into computing and engineering: from creating robust technologies for sensing, to 
actually incorporating such information as a part of advanced analytics and enhanced user experiences. As a Hispanic faculty, the PI serves as a mentor and role model for high school, undergraduate 
and graduate students involved in the Minority Scholars Symposium, Diversity Scholarship Program and Graduate Student Mentoring Program at the University of Texas at Dallas. Through lab open houses, demonstrations, and active online and social media presence, the PI is reaching out to non-traditional students, as well 
as the broader, non-technical audience interested in human behavior science.The project evaluates 
the powerful, scalable and appealing concept of using neutral reference models to contrast deviations in speech characteristics associated with emotions. The study proposes flexible, integrative and discriminative frameworks that capture the underlying encoding process of expressive behaviors including of emotion salient regions in the speech stream, intrinsic reliability of features,
 and dynamic evolution of emotions. The study considers binary and rank-based classifiers to recognize and rank-order specific expressive behaviors. The project presents speaker and lexical compensation schemes, and model adaptation strategies to increase the robustness of the proposed models. All these theoretical and algorithmic advances are carefully evaluated with naturalistic data, in which emotional content will be annotated 
with a novel crowdsourcing scheme that tracks in real time the performance of the evaluators.

识别和表征情绪行为是充实的，但非常重要的研究主题，用于丰富的语音分析和人类计算机的互动。这个职业项目旨在创建新颖的算法，以识别言语自发的情感行为，这些言语捕捉了情绪的基本外部化过程，并推广到在现实情况下收集的人类互动的记录。缺乏对当前语音情感算法识别自然人类互动期间表现性行为的概括是在现实生活应用中部署情感感知技术的关键障碍。在以表达行为的非均匀外部化为基础的理论框架下，该项目带来了转化解决方案来解决这一问题。拟议的模型和算法承诺见解语言＆＃8232;和副语言人类行为。可以出现几种新的科学途径，这些途径可以作为真正创新的进步，这些进步将影响安全和防御中的应用，下一代高级用户界面，健康行为信息学和教育。人类以人为中心的技术的作用，特别是在直接社会相关性的应用中进行背景化的角色，可以激发年轻的学者进入计算和工程学：从创建强大的传感技术，到实际上将这些信息作为高级分析和增强的用户体验的一部分。作为西班牙裔教师，PI是高中的导师和榜样，本科生以及参与了德克萨斯大学达拉斯大学的少数民族学者研讨会，多样性奖学金计划和研究生指导计划的研究生。通过实验室开放式房屋，演示以及积极的在线和社交媒体的存在，PI也与非传统学生联系，以及对人类行为科学感兴趣的广泛，非技术受众。该研究提出了灵活的，综合的和歧视性的框架，这些框架捕获了表达行为的基本编码过程，包括语音流中情绪突出区域，特征的内在可靠性，＆＃8232;和情绪的动态演变。该研究认为基于二进制和等级的分类器识别和等级特定的表达行为。该项目介绍了发言人和词汇补偿方案，以及模型适应策略，以提高所提出模型的鲁棒性。所有这些理论和算法的进步都通过自然主义数据进行了仔细的评估，其中将通过一种新型的众包方案来注释情感内容，该方案可以实时跟踪评估者的性能。

项目成果

Retrieving Speech Samples with Similar Emotional Content Using a Triplet Loss Function

• DOI: 10.1109/icassp.2019.8683273
• 发表时间: 2019-05
• 期刊: ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
• 作者: John Harvill;Mohammed Abdel-Wahab;Reza Lotfian;C. Busso

Audiovisual Speech Activity Detection with Advanced Long Short-Term Memory

• DOI: 10.21437/interspeech.2018-2490
• 发表时间: 2018-09
• 作者: Fei Tao;C. Busso

Generative Approach Using Soft-Labels to Learn Uncertainty in Predicting Emotional Attributes

使用软标签的生成方法来学习预测情感属性的不确定性

• DOI: 10.1109/acii52823.2021.9597461
• 发表时间: 2021
• 期刊: International Conference on Affective Computing and Intelligent Interaction (ACII 2021
• 作者: Sridhar, Kusha;Lin, Wei-Cheng;Busso, Carlos
• 通讯作者: Busso, Carlos

End-to-end audiovisual speech activity detection with bimodal recurrent neural models

使用双模循环神经模型进行端到端视听语音活动检测

• DOI: 10.1016/j.specom.2019.07.003
• 发表时间: 2019
• 期刊: Speech Communication
• 影响因子: 3.2
• 作者: Tao, Fei;Busso, Carlos
• 通讯作者: Busso, Carlos

Domain Adversarial for Acoustic Emotion Recognition

• DOI: 10.1109/taslp.2018.2867099
• 发表时间: 2018-12-01
• 期刊: IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING
• 影响因子: 5.4
• 作者: Abdelwahab, Mohammed;Busso, Carlos
• 通讯作者: Busso, Carlos