Collaborative Research: Monitoring Student State in Tutorial Spoken Dialogue

本研究探讨在口语对话辅导系统中监控学生情绪的可行性和实用性。虽然人类教师对学生话语的内容和潜在的感知情绪做出反应，但大多数辅导对话系统无法检测学生的情绪，而且是基于文本的，这可能会限制他们在情绪预测方面的成功。虽然在呼叫中心等口语对话应用程序中识别有问题的情绪(如沮丧、愤怒)的兴趣越来越大，但很少有工作涉及到辅导领域。PI正在调查词汇、句法、对话、韵律和声学线索的使用，以使计算机教师能够自动预测和回应学生的情绪。这项研究是在ITSPOKE的背景下进行的，ITSPOKE是一个基于语音的概念物理教学对话系统。PI正在记录学生与ITSPOKE的互动，在这些以及人与人对话中手动注释学生的情绪，识别注释的语言和副语言线索，并使用机器学习从潜在线索中预测情绪。然后，PI正在根据情绪识别得出调整系统辅导的策略。主要的科学贡献将是理解口语对话系统可用的线索是否可以用来预测情绪，并最终提高辅导性能。这一结果将对其他可以从监控情绪语音中受益的应用程序具有价值。在缩小人类导师和当前机器导师之间的性能差距方面取得的进展也将扩大当前计算机导师的有用性。

This research investigates the feasibility and utility of monitoring student emotions in spoken dialogue tutorial systems. While human tutors respond to both the content of student utterances and underlying perceived emotions, most tutorial dialogue systems cannot detect student emotions, and furthermore are text-based, which may limit their success at emotion prediction. While there has been increasing interest in identifying problematic emotions (e.g. frustration, anger) in spoken dialogue applications such as call centers, little work has addressed the tutorial domain.The PIs are investigating the use of lexical, syntactic, dialogue, prosodic and acoustic cues to enable a computer tutor to automatically predict and respond to student emotions. The research is being performed in the context of ITSPOKE, a speech-based tutoring dialogue system for conceptual physics. The PIs are recording students interacting with ITSPOKE, manually annotating student emotions in these as well as in human-human dialogues, identifying linguistic and paralinguistic cues to the annotations, and using machine learning to predict emotions from potential cues. The PIs are then deriving strategies for adapting the system's tutoring based upon emotion identification.The major scientific contribution will be an understanding of whether cues available to spoken dialogue systems can be used to predict emotion, and ultimately to improve tutoring performance. The results will be of value to other applications that can benefit from monitoring emotional speech. Progress towards closing the performance gap between human tutors and current machine tutors will also expand the usefulness of current computer tutors.