CHS: Small: Emotion-Aware Internet-of-Things Based on Analysis of Speech and Physiological Data

CHS：小型：基于语音和生理数据分析的情感感知物联网

• 批准号: 1908991
• 负责人: Christian Poellabauer
• 金额: $ 49.78万
• 依托单位: University of Notre Dame
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-09-01 至 2021-10-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1908991&HistoricalAwards=false
• 关键词: CHS; Small; Emotion; Aware; Internet

The Internet of Things describes a network of devices capable of connecting hundreds of billions of devices, and then sensing and communicating information required for a wide range of uses such as healthcare, vehicular systems, and industrial environments. As one of the most natural ways of communication, speech will increasingly be used as the primary form of interaction between humans and Internet of Things devices. In recent years, research has shown that there are clear links between the emotional and mental state of an individual and certain patterns in the individual's speech. If these patterns are detected in a timely fashion, it is possible to build emotion-aware Internet of Things solutions. This could be used to adapt a system to better meet the needs of the user, to prevent human error, to detect and prevent potentially malicious user activities, and to initiate medical interventions. Therefore, the overarching goal of this project is to advance speech-based emotion analysis to enable the design of such emotion-aware Internet of Things solutions. The project will also enrich the team's ongoing outreach and educational goals, including mentorship of minority and high-school students, revision of existing and development of new courses aligned with the research challenges in the project, and tight integration of research activities and undergraduate education.The technical challenges in the project are organized into three main thrusts. First, the project will develop and evaluate multi-modal emotion detection systems, where speech analysis is coupled with other physiological metrics such as heart rate, galvanic skin response, or skin temperature, to more accurately determine an individual's emotional state. Second, the work will apply the concept of topic modeling to perform context-aware analysis of speech data, which will also assist in differentiating short-term emotions (i.e., the current mood of an individual) from long-term emotions (e.g., depression). Topic modeling is an increasingly popular technique to learn, recognize, and extract the topics of spoken commands or conversations, providing additional context information for more accurate emotion analysis. The primary outcomes of the first two thrusts will be new insights into the design and development of emotion-aware systems. However, to achieve this goal, a comprehensive database containing speech and physiological data (annotated with the emotional states of the users) will be required, and therefore, the third thrust of the project will build such a database. When completed, this database will contain speech samples and other data from over 500 individuals and the database will be made available to the general scientific community to advance research beyond the team's institution.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

物联网描述了一种设备网络，能够连接数千亿台设备，然后感测和传输医疗保健、车辆系统和工业环境等广泛用途所需的信息。作为最自然的通信方式之一，语音将越来越多地被用作人类与物联网设备之间交互的主要形式。近年来，研究表明，一个人的情绪和精神状态与他说话的某些模式之间存在着明显的联系。如果及时检测到这些模式，就有可能构建情感感知的物联网解决方案。这可以用来调整系统，以更好地满足用户的需求，防止人为错误，检测和防止潜在的恶意用户活动，并启动医疗干预。因此，该项目的总体目标是推进基于语音的情感分析，以实现此类情感感知物联网解决方案的设计。该项目还将丰富该团队正在进行的推广和教育目标，包括少数民族和高中学生的导师，根据项目中的研究挑战修改现有课程并开发新课程，以及将研究活动与本科教育紧密结合。该项目中的技术挑战分为三个主要方面。首先，该项目将开发和评估多模态情感检测系统，其中语音分析与其他生理指标（如心率、皮肤电反应或皮肤温度）相结合，以更准确地确定个人的情感状态。其次，这项工作将应用主题建模的概念来执行语音数据的上下文感知分析，这也将有助于区分短期情绪（即，个体的当前情绪）与长期情绪（例如，抑郁症）。主题建模是一种越来越流行的技术，用于学习、识别和提取口头命令或对话的主题，为更准确的情感分析提供额外的上下文信息。前两项研究的主要成果将是对情绪感知系统的设计和开发的新见解。然而，为了实现这一目标，将需要一个包含语音和生理数据的综合数据库（注释用户的情绪状态），因此，该项目的第三个重点将建立这样一个数据库。完成后，该数据库将包含来自500多个人的语音样本和其他数据，该数据库将提供给一般科学界，以推动团队机构以外的研究。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

项目成果

Detecting Replay Attacks Using Multi-Channel Audio: A Neural Network-Based Method

• DOI: 10.1109/lsp.2020.2996908
• 发表时间: 2020-01-01
• 期刊: IEEE SIGNAL PROCESSING LETTERS
• 影响因子: 3.9
• 作者: Gong, Yuan;Yang, Jian;Poellabauer, Christian
• 通讯作者: Poellabauer, Christian

Experiences in Designing a Mobile Speech-Based Assessment Tool for Neurological Diseases

神经系统疾病移动语音评估工具的设计经验

• DOI: 10.1007/978-3-030-70569-5_1
• 发表时间: 2021
• 期刊: International Conference on Wireless Mobile Communication and Healthcare (MobiHealth
• 作者: Daudet, L.;Poellabauer, Christian;Schneider, Sandra
• 通讯作者: Schneider, Sandra

Design of a Neurocognitive Digital Health System (NDHS) for Neurodegenerative Diseases

针对神经退行性疾病的神经认知数字健康系统 (NDHS) 的设计

• DOI: 10.1145/3469266.3471157
• 发表时间: 2021
• 期刊: Workshop on Future of Digital Biomarkers
• 作者: Templeton, John M.;Poellabauer, Christian;Schneider, Sandra
• 通讯作者: Schneider, Sandra

Design of a Mobile-Based Neurological Assessment Tool for Aging Populations

针对老龄化人群的基于移动设备的神经评估工具的设计

• DOI: 10.1007/978-3-030-70569-5_11
• 发表时间: 2021
• 期刊: International Conference on Wireless Mobile Communication and Healthcare
• 作者: Templeton, John M;Poellabauer, Christian;Schneider, Sandra
• 通讯作者: Schneider, Sandra