CAREER: Advanced Knowledge Extraction of Affective Behaviors During Natural Human Interaction

识别和表征情绪行为是富语音衍生分析和人机交互的重要研究课题。这个CAREER项目旨在创建新的算法来识别从语音中自发的情感行为，捕捉情绪的潜在外化过程，并推广到在现实世界条件下收集的人类互动记录。目前缺乏泛化的语音情感算法来识别人类自然互动中的表达行为，这是在现实生活中应用情感感知技术的主要障碍。在基于表达行为的非统一外化的理论框架下，该项目为解决这一问题提供了变革性的解决方案。所提出的模型和算法有望探索和扩展语言学理论的见解。以及人类的副语言行为。一些新的科学途径可能会出现，作为真正的创新进步，将影响安全和国防、下一代高级用户界面、健康行为信息学和教育方面的应用。以人为本的技术的作用，特别是在与社会直接相关的应用中，可以激励年轻人。学者进入计算和工程：从创建强大的传感技术，到&amp；#8232；实际上，将这些信息作为高级分析和增强用户体验的一部分。作为一名西班牙裔教师，PI是高中和本科生的导师和榜样&amp；#8232；以及参加德克萨斯大学达拉斯分校少数民族学者研讨会、多样性奖学金计划和研究生指导计划的研究生。通过实验室开放日、演示以及活跃的在线和社交媒体活动，PI也在接触非传统学生&amp；#8232；作为对人类行为科学感兴趣的更广泛的非技术观众。项目评估&amp；#8232；使用中性参考模型来对比与情绪相关的语音特征偏差的强大，可扩展和吸引人的概念。该研究提出了灵活的、综合的和有区别的框架，这些框架捕捉了表达行为的潜在编码过程，包括语音流中的情绪显著区域、特征的内在可靠性和语言的编码过程。以及情绪的动态演变。该研究考虑了二元分类器和基于秩的分类器来识别和排序特定的表达行为。该项目提出了说话人和词汇补偿方案，以及模型自适应策略，以提高所提出模型的鲁棒性。所有这些理论和算法的进步都是用自然主义的数据仔细评估的，其中情感内容将被注释。采用一种新颖的众包方案，实时跟踪评估者的表现。

Identifying and characterizing emotional behaviors are challenging but very important research topics for enriched speech-derived analytics and human-computer interaction. This CAREER project aims to create novel algorithms to recognize spontaneous affective behaviors from speech that capture the underlying externalization process of emotions and generalize to recordings of human interactions collected under real-world conditions. The lack of generalization of current speech emotion algorithms to recognize expressive behaviors during natural human interaction is the key barrier to deploying affective-aware technology in real-life applications. Under a theoretical framework grounded in the nonuniform externalization of expressive behaviors, the project brings transformative solutions to address this problem. The proposed models and algorithms promise insights to explore and extend theories in linguistic
 and paralinguistic human behaviors. Several new scientific avenues can emerge that serve as truly innovative advancements that will impact applications in security and defense, next generation of advanced user interfaces, health behavior informatics, and education. The role of human centered technologies, especially contextualized in applications of direct societal relevance, can inspire young 
scholars into computing and engineering: from creating robust technologies for sensing, to 
actually incorporating such information as a part of advanced analytics and enhanced user experiences. As a Hispanic faculty, the PI serves as a mentor and role model for high school, undergraduate 
and graduate students involved in the Minority Scholars Symposium, Diversity Scholarship Program and Graduate Student Mentoring Program at the University of Texas at Dallas. Through lab open houses, demonstrations, and active online and social media presence, the PI is reaching out to non-traditional students, as well 
as the broader, non-technical audience interested in human behavior science.The project evaluates 
the powerful, scalable and appealing concept of using neutral reference models to contrast deviations in speech characteristics associated with emotions. The study proposes flexible, integrative and discriminative frameworks that capture the underlying encoding process of expressive behaviors including of emotion salient regions in the speech stream, intrinsic reliability of features,
 and dynamic evolution of emotions. The study considers binary and rank-based classifiers to recognize and rank-order specific expressive behaviors. The project presents speaker and lexical compensation schemes, and model adaptation strategies to increase the robustness of the proposed models. All these theoretical and algorithmic advances are carefully evaluated with naturalistic data, in which emotional content will be annotated 
with a novel crowdsourcing scheme that tracks in real time the performance of the evaluators.