EAGER:Towards Adaptive and Robust Multimodal Emotion Recognition In-the-Wild

EAGER:迈向自适应且鲁棒的野外多模态情绪识别

基本信息

  • 批准号:
    2034791
  • 负责人:
  • 金额:
    $ 9.99万
  • 依托单位:
  • 依托单位国家:
    美国
  • 项目类别:
    Standard Grant
  • 财政年份:
    2020
  • 资助国家:
    美国
  • 起止时间:
    2020-08-01 至 2023-07-31
  • 项目状态:
    已结题

项目摘要

Emotions are essential to human life. They directly influence human perception and behaviors, and have big impacts on people's daily tasks, such as learning, social interaction, and decision-making. Automatic emotion recognition has found applications in many domains such as human-computer interaction, human-robot interaction, multimedia retrieval, social media analysis, and healthcare. Emotional states are expressed through a variety of channels including facial expression, voice prosody, spoken words, and body gestures. Automatic emotion recognition in real-world applications is a challenging task. Real-world emotions involve subtle expressive behaviors, different degrees of expressiveness in different channels, and the imperfect conditions such as background noise or music, poor illumination, and uncontrolled head poses for example. This EArly-concept Grant for Exploratory Research project aims to address the challenges of spontaneous emotion expressions and imperfect audio and video signals in-the-wild, and develop a novel multimodal emotion recognition system for real-world applications. The research will lead to advances in data collection, algorithm design, and bench-marking for the next generation of affective computing.This project consists of several research components. First, a multimodal dataset of spontaneous emotion expressions in-the-wild will be developed. The dataset will contain natural spontaneous emotion data in various challenging real life environments, and crowd-sourced ratings in different modalities (audio and video channels). A thorough benchmark analysis using this dataset will be conducted tostudy how different features, modalities, and signal impairments contribute to the success and failure of emotion recognition systems. Finally, novel multimodal emotion recognition algorithms will be designed using adaptive and robust multimodal learning and fusion. The research findings will be made available through dataset sharing, publications, talks, and open-source codes, allowing a multitude of developers, researchers, and companies to improve and evolve multimodal emotion recognition in real-world applications. The project will also provide novel research opportunities for graduate and undergraduate students, including women and minority students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
情感对人类生活是必不可少的。它们直接影响人类的感知和行为,并对人们的日常任务,如学习,社会互动和决策产生重大影响。自动情感识别在人机交互、人机交互、多媒体检索、社会媒体分析和医疗保健等领域得到了广泛的应用。情绪状态通过多种渠道表达,包括面部表情、声音韵律、言语和肢体动作。在现实应用中,自动情绪识别是一项具有挑战性的任务。现实世界的情绪包括微妙的表达行为,不同渠道的不同程度的表达,以及不完美的条件,如背景噪音或音乐,光线不足,不受控制的头部姿势。这个探索性研究项目的早期概念拨款旨在解决自然情绪表达和不完善的音频和视频信号的挑战,并开发一种新的多模态情绪识别系统,用于现实世界的应用。这项研究将为下一代情感计算带来数据收集、算法设计和基准测试方面的进步。这个项目由几个研究部分组成。首先,将开发一个自然情绪表达的多模态数据集。该数据集将包含各种具有挑战性的现实生活环境中的自然自发情绪数据,以及不同模式(音频和视频频道)的众包评级。使用此数据集进行全面的基准分析,以研究不同的特征,模式和信号损伤如何影响情绪识别系统的成功和失败。最后,利用自适应和鲁棒的多模态学习和融合设计新的多模态情感识别算法。研究结果将通过数据集共享、出版物、演讲和开源代码提供,允许众多开发人员、研究人员和公司在现实世界的应用中改进和发展多模态情感识别。该项目还将为研究生和本科生,包括女性和少数民族学生提供新的研究机会。该奖项反映了美国国家科学基金会的法定使命,并通过使用基金会的知识价值和更广泛的影响审查标准进行评估,被认为值得支持。

项目成果

期刊论文数量(4)
专著数量(0)
科研奖励数量(0)
会议论文数量(0)
专利数量(0)
Analysis of Eye Fixations During Emotion Recognition in Talking Faces
说话面孔情绪识别过程中眼睛注视的分析
Audio-Visual Emotion Recognition With Preference Learning Based on Intended and Multi-Modal Perceived Labels
Multimodal Emotion Recognition with Surgical and Fabric Masks
使用外科口罩和织物口罩进行多模式情绪识别
  • DOI:
    10.1109/icassp43922.2022.9746414
  • 发表时间:
    2022
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Yang, Ziqing;Nayan, Katherine;Fan, Zehao;Cao, Houwei
  • 通讯作者:
    Cao, Houwei
Exploration of Acoustic and Lexical Cues for the INTERSPEECH 2020 Computational Paralinguistic Challenge
INTERSPEECH 2020 计算副语言挑战赛的声学和词汇线索探索
  • DOI:
    10.21437/interspeech.2020-2999
  • 发表时间:
    2020
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Yang, Ziqing;An, Zifan;Fan, Zehao;Jing, Chengye;Cao, Houwei
  • 通讯作者:
    Cao, Houwei
{{ item.title }}
{{ item.translation_title }}
  • DOI:
    {{ item.doi }}
  • 发表时间:
    {{ item.publish_year }}
  • 期刊:
  • 影响因子:
    {{ item.factor }}
  • 作者:
    {{ item.authors }}
  • 通讯作者:
    {{ item.author }}

数据更新时间:{{ journalArticles.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ monograph.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ sciAawards.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ conferencePapers.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ patent.updateTime }}

Houwei Cao其他文献

Deriving MFCC Parameters from the Dynamic Spectrum for Robust Speech Recognition
从动态频谱中导出 MFCC 参数以实现稳健的语音识别
Emerging Evidence for Automatic Acoustic Analysis as a Predictor of Severity of Parkinson’s Disease
  • DOI:
    10.1016/j.apmr.2015.08.252
  • 发表时间:
    2015-10-01
  • 期刊:
  • 影响因子:
  • 作者:
    Houwei Cao;Linda Tickle-Degnen;Matthias Scheutz
  • 通讯作者:
    Matthias Scheutz
Combining Ranking and Classification to Improve Emotion Recognition in Spontaneous Speech
结合排序和分类来提高自发言语中的情绪识别
  • DOI:
  • 发表时间:
    2012
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Houwei Cao;R. Verma;A. Nenkova
  • 通讯作者:
    A. Nenkova
Realtime mobile bandwidth prediction using LSTM neural network and Bayesian fusion
  • DOI:
    10.1016/j.comnet.2020.107515
  • 发表时间:
    2020-12-09
  • 期刊:
  • 影响因子:
  • 作者:
    Lifan Mei;Runchen Hu;Houwei Cao;Yong Liu;Zifan Han;Feng Li;Jin Li
  • 通讯作者:
    Jin Li
Semantics-based language modeling for Cantonese-English code-mixing speech recognition
基于语义的粤语-英语混码语音识别语言建模

Houwei Cao的其他文献

{{ item.title }}
{{ item.translation_title }}
  • DOI:
    {{ item.doi }}
  • 发表时间:
    {{ item.publish_year }}
  • 期刊:
  • 影响因子:
    {{ item.factor }}
  • 作者:
    {{ item.authors }}
  • 通讯作者:
    {{ item.author }}

相似海外基金

CAREER: Adaptive Deep Learning Systems Towards Edge Intelligence
职业:迈向边缘智能的自适应深度学习系统
  • 批准号:
    2338512
  • 财政年份:
    2024
  • 资助金额:
    $ 9.99万
  • 项目类别:
    Continuing Grant
CAREER: Towards Fundamentals of Adaptive, Collaborative and Intelligent Radar Sensing and Perception
职业:探索自适应、协作和智能雷达传感和感知的基础知识
  • 批准号:
    2340029
  • 财政年份:
    2024
  • 资助金额:
    $ 9.99万
  • 项目类别:
    Continuing Grant
CAREER: Towards Environment-Aware Adaptive Safety for Learning-Enabled Multiagent Systems with Application to Target Drone Capturing
职业:为支持学习的多智能体系统实现环境感知的自适应安全,并应用于目标无人机捕获
  • 批准号:
    2336189
  • 财政年份:
    2024
  • 资助金额:
    $ 9.99万
  • 项目类别:
    Continuing Grant
SINCERE The second life of modern period architecture: Resilient and adaptive renovation towards net-zero carbon
SINCERE 现代建筑的第二次生命:弹性和适应性改造,实现净零碳
  • 批准号:
    10089449
  • 财政年份:
    2023
  • 资助金额:
    $ 9.99万
  • 项目类别:
    EU-Funded
CAREER: Adaptive experiments towards learning treatment effect heterogeneity
职业:学习治疗效果异质性的适应性实验
  • 批准号:
    2239047
  • 财政年份:
    2023
  • 资助金额:
    $ 9.99万
  • 项目类别:
    Continuing Grant
SINCERE - The second life of modern period architecture: Resilient and adaptive renovation towards net-zero carbon heritage buildings
SINCERE - 现代建筑的第二次生命:对净零碳遗产建筑进行弹性和适应性改造
  • 批准号:
    10092236
  • 财政年份:
    2023
  • 资助金额:
    $ 9.99万
  • 项目类别:
    EU-Funded
Development of a mentalizing-based adaptive support model focusing on attitudes towards ambiguity
开发基于心智化的自适应支持模型,重点关注对模糊性的态度
  • 批准号:
    23K02928
  • 财政年份:
    2023
  • 资助金额:
    $ 9.99万
  • 项目类别:
    Grant-in-Aid for Scientific Research (C)
Adaptive support for diverse class styles using ICT towards achieving educational DX
利用 ICT 对不同班级风格提供适应性支持,以实现教育 DX
  • 批准号:
    23K02660
  • 财政年份:
    2023
  • 资助金额:
    $ 9.99万
  • 项目类别:
    Grant-in-Aid for Scientific Research (C)
ERI: Towards Intelligent, Cost-Efficient, and Adaptive Techniques to Enable Biomedical Hardware-Assisted Cybersecurity
ERI:采用智能、经济高效和自适应的技术来实现生物医学硬件辅助的网络安全
  • 批准号:
    2139034
  • 财政年份:
    2022
  • 资助金额:
    $ 9.99万
  • 项目类别:
    Standard Grant
Towards a new generation of adaptive climate and weather models
迈向新一代自适应气候和天气模型
  • 批准号:
    RGPIN-2018-05586
  • 财政年份:
    2022
  • 资助金额:
    $ 9.99万
  • 项目类别:
    Discovery Grants Program - Individual
{{ showInfoDetail.title }}

作者:{{ showInfoDetail.author }}

知道了