RI: Small: Advancing the Science of Generalizable and Personalizable Speech-Centered Self-Report Emotion Classifiers

该项目的目标是创建新的个性化语音情感识别方法，并使用这些方法来研究情绪变化与心理健康变化之间的关系。第一步是准确测量一个人的情绪在一天、一周、一个月甚至一年的时间里是如何变化的。然而，目前唯一可行的方法是每天多次主动询问用户的感受。用户通常愿意在较短的时间内这样做，但如果时间较长，这可能会很费力。幸运的是，语音数据通常很容易捕获并传达情感信息。然而，大多数语音情感识别方法并不关注用户的感受，而是关注于预测外部人群如何标记用户的感受。该项目的目标是将自动情绪分类重新聚焦于用户本身。在未来，这将使我们能够轻松地收集有关用户情绪的信息，从而对情绪变化与健康变化的风险因素之间的关系进行新的调查。提出的研究目标的目标是推进最先进的鲁棒和可推广的个性化语音（声学+语言）自我报告情绪识别分类器，并调查如何使用这些分类器创建的措施将允许研究人员在有自杀风险的个体的临床人群中直观地了解心理健康症状严重程度的变化。自动语音情感识别领域几乎完全专注于估计外部观察者如何感知给定的情感表现（即，感知他人）。然而，当重点放在这项技术的最终用例上时，例如，心理健康症状严重程度跟踪，这通常不是所需要的。相反，症状严重程度跟踪通常需要关于特定个体如何解释自己的情绪体验（即自我报告）的信息。例如，自我报告模式的改变与抑郁严重程度的变化有关。然而，这些变化目前只能通过积极参与来衡量，其中个人经常被要求使用自我报告方法（例如，生态瞬间评估，EMA）纵向描述他们的情感体验，每天多次，这在成本和参与者负担方面都是相当昂贵的。项目团队设想，未来音频可以被动收集并用于自动推断自我报告的情绪，但由于准确估计自我报告的情绪存在持续的挑战，包括认知偏见、环境以及自我报告和情绪体验之间的差异，因此对这种分类器的设计关注有限。项目团队将通过以下方式实现这些目标：1)使用新的指标创建鲁棒性和可泛化的分类器，这些指标鼓励模型像人类观察者一样关注相同的声音和语言线索；2)纵向为用户个性化分类器；3)利用现有的真实世界数据集，通过预测心理健康症状严重程度（自杀风险）的变化，评估自我报告情绪分类器的有效性。所提出的方法将推动调查如何使用被动收集的音频数据来估计健康变化的风险因素的变化。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The goal of the project is to create new and personalized speech emotion recognition approaches and to use these approaches to investigate how changes in emotion are related to changes in mental health. The first step is accurately measuring how a person’s emotions vary over the course of a day, a week, a month, or even a year. However, the only approaches currently available to do so involve actively asking a user how they feel multiple times per day. Users are often willing to do this over shorter periods of time, but over longer periods of time this can be quite taxing. Fortunately, speech data are often easy to capture and conveys information about emotion. However, most approaches in speech emotion recognition are not focused on how the user feels and instead are focused on predicting how an outside group of people would label that user’s feeling. The goal of the project is to refocus automatic emotion classification on the user themselves. In the future, this will allow us to easily collect information about a user’s emotion leading to new investigations into how changes in emotions are associated with risk factors for changes in health.The goal of the presented research objectives is to advance the state-of-the-art in robust and generalizable personalized speech (acoustics + language) self-report emotion recognition classifiers and to investigate how measures created using these classifiers will allow researchers to intuit changes in mental health symptom severity in a clinical population of individuals at risk for suicidality. The field of automatic speech emotion recognition is almost exclusively focused on estimating how an outside group of observers would perceive a given emotional display (i.e., perception-of-other). Yet, when the focus is on the ultimate use cases of this technology, e.g., mental health symptom severity tracking, this is often not what is needed. Instead, symptom severity tracking often needs information about how a given individual is interpreting their own emotional experiences (i.e., self-report). For example, changing patterns in self-report are associated with changes in depression severity. Yet, these changes are currently measurable only through active participation, in which individuals are regularly asked to describe their emotional experiences using self-report measures (e.g., Ecological Momentary Assessment, EMA) longitudinally, multiple times per day, which can be quite expensive both in terms of cost and participant burden. The project team envisions a future in which audio can be passively collected and used to automatically infer self-reported emotion, but there has been limited attention to the design of such classifiers due to persistent challenges associated with accurately estimating self-reported emotion, including cognitive bias, context, and the difference between self-report and emotional experiences. The project team will accomplish these goals by: 1) creating classifiers that are robust and generalizable using new metrics that encourage models to attend to the same acoustic and language cues as human observers; 2) personalizing classifiers to users longitudinally, and 3) evaluating the effectiveness of self-report emotion classifiers by predicting changes in mental health symptom severity using an existing real-world dataset annotated with mental health symptom severity (risk of suicide). The presented approaches will forward investigations into how to use passively collected audio data to estimate changes in risk factors for health changes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.