CAREER: Advanced Knowledge Extraction of Affective Behaviors During Natural Human Interaction

职业：人类自然互动过程中情感行为的高级知识提取

• 批准号: 1453781
• 负责人: Carlos Busso
• 金额: $ 49.59万
• 依托单位: University of Texas at Dallas
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-09-01 至 2021-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1453781&HistoricalAwards=false
• 关键词: CAREER; Advanced; Knowledge; Extraction; Affective

Identifying and characterizing emotional behaviors are challenging but very important research topics for enriched speech-derived analytics and human-computer interaction. This CAREER project aims to create novel algorithms to recognize spontaneous affective behaviors from speech that capture the underlying externalization process of emotions and generalize to recordings of human interactions collected under real-world conditions. The lack of generalization of current speech emotion algorithms to recognize expressive behaviors during natural human interaction is the key barrier to deploying affective-aware technology in real-life applications. Under a theoretical framework grounded in the nonuniform externalization of expressive behaviors, the project brings transformative solutions to address this problem. The proposed models and algorithms promise insights to explore and extend theories in linguistic
 and paralinguistic human behaviors. Several new scientific avenues can emerge that serve as truly innovative advancements that will impact applications in security and defense, next generation of advanced user interfaces, health behavior informatics, and education. The role of human centered technologies, especially contextualized in applications of direct societal relevance, can inspire young 
scholars into computing and engineering: from creating robust technologies for sensing, to 
actually incorporating such information as a part of advanced analytics and enhanced user experiences. As a Hispanic faculty, the PI serves as a mentor and role model for high school, undergraduate 
and graduate students involved in the Minority Scholars Symposium, Diversity Scholarship Program and Graduate Student Mentoring Program at the University of Texas at Dallas. Through lab open houses, demonstrations, and active online and social media presence, the PI is reaching out to non-traditional students, as well 
as the broader, non-technical audience interested in human behavior science.The project evaluates 
the powerful, scalable and appealing concept of using neutral reference models to contrast deviations in speech characteristics associated with emotions. The study proposes flexible, integrative and discriminative frameworks that capture the underlying encoding process of expressive behaviors including of emotion salient regions in the speech stream, intrinsic reliability of features,
 and dynamic evolution of emotions. The study considers binary and rank-based classifiers to recognize and rank-order specific expressive behaviors. The project presents speaker and lexical compensation schemes, and model adaptation strategies to increase the robustness of the proposed models. All these theoretical and algorithmic advances are carefully evaluated with naturalistic data, in which emotional content will be annotated 
with a novel crowdsourcing scheme that tracks in real time the performance of the evaluators.

识别和表征情绪行为是一个具有挑战性但非常重要的研究课题，用于丰富的语音衍生分析和人机交互。这个CAREER项目旨在创建新的算法来识别来自语音的自发情感行为，这些语音捕获了情感的潜在外部化过程，并推广到在真实世界条件下收集的人类互动记录。目前的语音情感算法缺乏泛化能力来识别自然人类交互过程中的表达行为，这是将情感感知技术应用于现实生活中的关键障碍。在基于表达行为的非统一外部化的理论框架下，该项目为解决这一问题带来了变革性的解决方案。所提出的模型和算法承诺的见解，探索和扩展理论的语言#8232;和非语言人类行为。一些新的科学途径可以作为真正的创新进步出现，这些进步将影响安全和国防，下一代高级用户界面，健康行为信息学和教育领域的应用。以人为本的技术的作用，特别是在直接社会相关性的应用中，可以激励年轻的学者进入计算和工程领域：从创建强大的传感技术，到实际将这些信息作为高级分析和增强用户体验的一部分。作为一个西班牙裔教师，PI作为高中，本科
和研究生参与少数民族学者研讨会，多样性奖学金计划和研究生导师计划在得克萨斯大学达拉斯。通过实验室开放日、演示以及活跃的在线和社交媒体活动，PI正在接触非传统的学生，&以及对人类行为科学感兴趣的更广泛的非技术受众&。使用中性参考模型来对比与情绪相关的语音特征偏差的强大、可扩展和吸引人的概念。该研究提出了灵活的，综合的和有区别的框架，捕捉表达行为的底层编码过程，包括语音流中的情感突出区域，特征的内在可靠性，以及情感的动态演变。该研究认为，二进制和排名为基础的分类识别和排名的具体表达行为。该项目提出了扬声器和词汇补偿计划，模型自适应策略，以增加所提出的模型的鲁棒性。所有这些理论和算法的进步都是用自然主义数据仔细评估的，其中情感内容将用一种新颖的众包方案进行注释，该方案可以真实的时间跟踪评估者的表现。

项目成果

Preference-Learning with Qualitative Agreement for Sentence Level Emotional Annotations

句子级情感注释的定性一致性偏好学习

• DOI: 10.21437/interspeech.2018-2478
• 发表时间: 2018
• 期刊: Interspeech 2018
• 作者: Parthasarathy, Srinivas;Busso, Carlos
• 通讯作者: Busso, Carlos

Retrieving Speech Samples with Similar Emotional Content Using a Triplet Loss Function

• DOI: 10.1109/icassp.2019.8683273
• 发表时间: 2019-05
• 期刊: ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
• 作者: John Harvill;Mohammed Abdel-Wahab;Reza Lotfian;C. Busso

Audiovisual Speech Activity Detection with Advanced Long Short-Term Memory

• DOI: 10.21437/interspeech.2018-2490
• 发表时间: 2018-09
• 作者: Fei Tao;C. Busso

Domain Adversarial for Acoustic Emotion Recognition

• DOI: 10.1109/taslp.2018.2867099
• 发表时间: 2018-12-01
• 期刊: IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING
• 影响因子: 5.4
• 作者: Abdelwahab, Mohammed;Busso, Carlos
• 通讯作者: Busso, Carlos

Generative Approach Using Soft-Labels to Learn Uncertainty in Predicting Emotional Attributes

使用软标签的生成方法来学习预测情感属性的不确定性

• DOI: 10.1109/acii52823.2021.9597461
• 发表时间: 2021
• 期刊: International Conference on Affective Computing and Intelligent Interaction (ACII 2021
• 作者: Sridhar, Kusha;Lin, Wei-Cheng;Busso, Carlos
• 通讯作者: Busso, Carlos