RI: Small: An Affect-Adaptive Spoken Dialogue System that Responds Based on User Model and Multiple Affective States

人们对情感对话系统越来越感兴趣，因为人们相信，在人与人之间的对话中，参与者似乎(至少在一定程度上)对其他参与者的情绪、态度和元认知状态做出了反应。这项研究的目标是通过借鉴先前在更广泛的口语对话和情感系统社区中的研究结果，从三个维度改善情感口语对话系统的技术水平。首先，先前的研究表明，并不是所有的用户都以相同的方式与系统交互；拟议的研究假设，对不同领域能力水平的用户采用不同的情感适应将进一步提高情感口语对话系统的绩效。其次，先前的研究表明，用户在与系统交互时表现出一系列的情感状态和态度；拟议的研究假设，适应多个用户状态将在情感口语对话系统中产生进一步的性能改善。第三，虽然先前的研究已经表明，在半自动对话系统中，情感适应的初步性能改善，但在全自动系统中尚未实现类似的改善。这项拟议的研究将使用最先进的经验方法来建立全自动的情感探测器。假设对话系统的全自动版本和半自动版本都可以比非自适应对话系统提高性能，其中半自动产生的改进最大。这三个假设将在现有口语对话辅导系统的背景下进行调查，该系统适应用户的不确定状态。任务领域是概念物理，通常在一年级的物理课程中涵盖(例如，牛顿定律、引力等)。为了研究第一个假设，将开发第一个增强的系统版本；它将使用现有的针对低能力用户的领域知识的不确定性适应，并将开发和实施新的不确定性适应，以用于更高能力的用户。为了研究第二个假设，将开发第二个增强的系统版本；它将使用现有的不确定性适应来处理所有显示不确定性的话轮，并将开发和实施新的脱离适应，以应用于所有显示第二脱离状态的学生话轮。然后将在绿野仙踪(WOZ)设置中进行两个增强型系统的对照实验，由人类向导检测情感并执行语音识别和语言理解。为了研究第三个假设，将进行第二个对照实验，用全自动系统取代WOZ系统版本。这项研究的主要智力贡献将是证明在部分和全自动情感口语对话教学系统中是否可以实现显著的性能提升：1)通过适应基于用户能力水平的用户不确定性，以及2)通过适应在辅导领域中最重要的多个用户状态，即不确定性和脱离。因此，该研究项目将推动口语对话和计算机辅导技术的发展，同时展示理想条件下与现实条件下情感适应系统的任何不同效果。更广泛地说，这项研究和由此产生的技术将导致更自然和更有效的基于口语对话的系统，无论是用于辅导还是用于更传统的信息寻求领域。此外，提高计算机教师的绩效将扩大他们的有用性，从而对教育和社会产生实质性的好处。

There has been increasing interest in affective dialogue systems, motivated by the belief that in human-human dialogues, participants seem to be (at least to some degree) detecting and responding to the emotions, attitudes and metacognitive states of other participants. The goal of the proposed research is to improve the state of the art in affective spoken dialogue systems along three dimensions, by drawing on the results of prior research in the wider spoken dialogue and affective system communities. First, prior research hasshown that not all users interact with a system in the same way; the proposed research hypothesizes that employing different affect adaptations for users with different domain aptitude levels will yield further performance improvement in affective spoken dialogue systems. Second, prior research has shown that users display a range of affective states and attitudes while interacting with a system; the proposed research hypothesizes that adapting to multiple user states will yield further performance improvement in affective spoken dialogue systems. Third, while prior research has shown preliminary performance gains for affect adaptation in semi-automated dialogue systems, similar gains have not yet been realized in fully automated systems. The proposed research will use state of the art empirical methods to build fully automated affect detectors. It is hypothesized that both fully and semi-automated versions of a dialogue systemthat either adapts to affect differently depending on user class, or that adapts to multiple user affective states, can improve performance compared to non-adaptive counterparts, with semi-automation generating the most improvement. The three hypotheses will be investigated in the context of an existing spoken dialogue tutoring system that adapts to the user state of uncertainty. The task domain is conceptual physics typically covered in a first-year physics course (e.g., Newtons Laws, gravity, etc.). To investigate the first hypothesis, a first enhanced system version will be developed; it will use the existing uncertainty adaptation for lower aptitude users with respect to domain knowledge, and a new uncertainty adaptation will be developed and implemented to be employed for higher aptitude users. To investigate the second hypothesis, a second enhanced systemversion will be developed; it will use the existing uncertainty adaptation for all turns displaying uncertainty, and a new disengagement adaptation will be developed and implemented to be employed for all student turns displaying a second state of disengagement. A controlled experiment with the two enhanced systems will then be conducted in a Wizard-of-Oz (WOZ) setup, with a human Wizard detecting affect and performing speech recognition and language understanding. To investigate the third hypothesis, a second controlled experiment will be conducted, which replaces the WOZ system versions with fully-automated systems.The major intellectual contribution of this research will be to demonstrate whether significant performance gains can be achieved in both partially and fully-automated affective spoken dialogue tutoring systems 1) by adapting to user uncertainty based on user aptitude levels, and 2) by adapting to multiple user states hypothesized to be of primary importance within the tutoring domain, namely uncertainty and disengagement. The research project will thus advance the state of the art in both spoken dialogue and computer tutoring technologies, while at the same time demonstrating any differing effects of affect-adaptive systems under ideal versus realistic conditions. More broadly, the research and resulting technology will lead to more natural and effective spoken dialogue-based systems, both for tutoring as well as for more traditional information-seeking domains. In addition, improving the performance of computer tutors will expand their usefulness and thus have substantial benefits for education and society.