CAREER: Advanced Knowledge Extraction of Affective Behaviors During Natural Human Interaction

识别和表征情绪行为是丰富的语音分析和人机交互的重要研究课题，具有挑战性。这个职业项目旨在创造新的算法，从语音中识别自发的情感行为，捕捉潜在的情感外化过程，并将其推广到在现实世界条件下收集的人类互动记录。当前的语音情感算法缺乏对自然人类交互过程中表达行为的识别的通用性，这是将情感感知技术应用于现实生活的关键障碍。在基于表达行为不一致外化的理论框架下，该项目提出了解决这一问题的变革性解决方案。所提出的模型和算法有望探索和扩展语言学和副语言人类行为的理论。可能会出现几种新的科学途径，作为真正的创新进步，将影响安全和防御、下一代高级用户界面、健康行为信息学和教育方面的应用。以人为中心的技术的作用，特别是在具有直接社会相关性的应用程序中的作用，可以激励年轻的学者进入计算和工程：从创建强大的传感技术，到实际将此类信息作为高级分析和增强的用户体验的一部分。作为一名西班牙裔教职员工，PI是德克萨斯大学达拉斯分校少数民族学者研讨会、多元化奖学金计划和研究生指导计划的高中、本科生和研究生的导师和榜样。通过实验室开放参观、演示以及活跃的在线和社交媒体展示，PI正在接触非传统学生，以及对人类行为科学感兴趣的更广泛的非技术受众。该项目评估了使用中性参考模型来对比与情绪相关的言语特征偏差的强大、可扩展和有吸引力的概念。这项研究提出了灵活的、综合的和区分的框架，该框架捕捉了表达行为的潜在编码过程，包括语音流中情绪显著区域的编码过程、特征的内在可靠性以及情绪的动态演变。这项研究认为二元分类器和基于等级的分类器识别和排名特定的表达行为。该项目提出了说话人和词汇补偿方案，以及模型自适应策略，以增加所提出模型的稳健性。所有这些理论和算法的进步都用自然数据进行了仔细的评估，其中情感内容将用一种新的众包方案进行注释，该方案实时跟踪评估者的表现。

Identifying and characterizing emotional behaviors are challenging but very important research topics for enriched speech-derived analytics and human-computer interaction. This CAREER project aims to create novel algorithms to recognize spontaneous affective behaviors from speech that capture the underlying externalization process of emotions and generalize to recordings of human interactions collected under real-world conditions. The lack of generalization of current speech emotion algorithms to recognize expressive behaviors during natural human interaction is the key barrier to deploying affective-aware technology in real-life applications. Under a theoretical framework grounded in the nonuniform externalization of expressive behaviors, the project brings transformative solutions to address this problem. The proposed models and algorithms promise insights to explore and extend theories in linguistic
 and paralinguistic human behaviors. Several new scientific avenues can emerge that serve as truly innovative advancements that will impact applications in security and defense, next generation of advanced user interfaces, health behavior informatics, and education. The role of human centered technologies, especially contextualized in applications of direct societal relevance, can inspire young 
scholars into computing and engineering: from creating robust technologies for sensing, to 
actually incorporating such information as a part of advanced analytics and enhanced user experiences. As a Hispanic faculty, the PI serves as a mentor and role model for high school, undergraduate 
and graduate students involved in the Minority Scholars Symposium, Diversity Scholarship Program and Graduate Student Mentoring Program at the University of Texas at Dallas. Through lab open houses, demonstrations, and active online and social media presence, the PI is reaching out to non-traditional students, as well 
as the broader, non-technical audience interested in human behavior science.The project evaluates 
the powerful, scalable and appealing concept of using neutral reference models to contrast deviations in speech characteristics associated with emotions. The study proposes flexible, integrative and discriminative frameworks that capture the underlying encoding process of expressive behaviors including of emotion salient regions in the speech stream, intrinsic reliability of features,
 and dynamic evolution of emotions. The study considers binary and rank-based classifiers to recognize and rank-order specific expressive behaviors. The project presents speaker and lexical compensation schemes, and model adaptation strategies to increase the robustness of the proposed models. All these theoretical and algorithmic advances are carefully evaluated with naturalistic data, in which emotional content will be annotated 
with a novel crowdsourcing scheme that tracks in real time the performance of the evaluators.