CHS: Small: Emotion-Aware Internet-of-Things Based on Analysis of Speech and Physiological Data

物联网描述了一个设备网络，能够连接数千亿台设备，然后感知和通信医疗保健、车辆系统和工业环境等广泛用途所需的信息。作为最自然的交流方式之一，语音将越来越多地被用作人类与物联网设备之间交互的主要形式。近年来，研究表明，个人的情绪和精神状态与个人言语中的某些模式之间存在明显的联系。如果及时发现这些模式，就有可能构建情感感知的物联网解决方案。这可用于调整系统以更好地满足用户的需求、防止人为错误、检测和防止潜在的恶意用户活动以及发起医疗干预。因此，本项目的总体目标是推进基于语音的情感分析，以使此类情感感知物联网解决方案的设计成为可能。该项目还将丰富该团队正在进行的外联和教育目标，包括指导少数族裔和高中生，根据项目中的研究挑战修订和开发现有课程和开发新课程，以及将研究活动和本科教育紧密结合。项目中的技术挑战被组织为三个主要方面。首先，该项目将开发和评估多模式情绪检测系统，其中语音分析与其他生理指标相结合，如心率、皮肤电反应或皮肤温度，以更准确地确定个人的情绪状态。其次，这项工作将应用主题建模的概念来对语音数据进行上下文感知分析，这也将有助于区分短期情绪(即，个人的当前情绪)和长期情绪(例如，抑郁)。主题建模是一种越来越流行的技术，用于学习、识别和提取口头命令或对话的主题，为更准确的情感分析提供额外的上下文信息。前两项努力的主要成果将是对情感感知系统的设计和开发的新见解。然而，为了实现这一目标，将需要一个全面的数据库，其中包含语音和生理数据(用用户的情绪状态进行注释)，因此，该项目的第三个推力将建立这样一个数据库。完成后，这个数据库将包含来自500多个人的语音样本和其他数据，该数据库将向普通科学界提供，以推动团队研究所以外的研究。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The Internet of Things describes a network of devices capable of connecting hundreds of billions of devices, and then sensing and communicating information required for a wide range of uses such as healthcare, vehicular systems, and industrial environments. As one of the most natural ways of communication, speech will increasingly be used as the primary form of interaction between humans and Internet of Things devices. In recent years, research has shown that there are clear links between the emotional and mental state of an individual and certain patterns in the individual's speech. If these patterns are detected in a timely fashion, it is possible to build emotion-aware Internet of Things solutions. This could be used to adapt a system to better meet the needs of the user, to prevent human error, to detect and prevent potentially malicious user activities, and to initiate medical interventions. Therefore, the overarching goal of this project is to advance speech-based emotion analysis to enable the design of such emotion-aware Internet of Things solutions. The project will also enrich the team's ongoing outreach and educational goals, including mentorship of minority and high-school students, revision of existing and development of new courses aligned with the research challenges in the project, and tight integration of research activities and undergraduate education.The technical challenges in the project are organized into three main thrusts. First, the project will develop and evaluate multi-modal emotion detection systems, where speech analysis is coupled with other physiological metrics such as heart rate, galvanic skin response, or skin temperature, to more accurately determine an individual's emotional state. Second, the work will apply the concept of topic modeling to perform context-aware analysis of speech data, which will also assist in differentiating short-term emotions (i.e., the current mood of an individual) from long-term emotions (e.g., depression). Topic modeling is an increasingly popular technique to learn, recognize, and extract the topics of spoken commands or conversations, providing additional context information for more accurate emotion analysis. The primary outcomes of the first two thrusts will be new insights into the design and development of emotion-aware systems. However, to achieve this goal, a comprehensive database containing speech and physiological data (annotated with the emotional states of the users) will be required, and therefore, the third thrust of the project will build such a database. When completed, this database will contain speech samples and other data from over 500 individuals and the database will be made available to the general scientific community to advance research beyond the team's institution.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.