CHS: Medium: Adapting to Affect in Multimodal Dialogue-Rich Interaction with Middle School Students

情感，或情感，深刻地塑造了人类的体验。它影响我们如何执行任务，如何与他人建立关系，以及如何驾驭日常生活的复杂性。情感受到与他人的交流、与自然世界的体验以及与机器的互动的塑造和影响。情感在学习中的作用尤为突出。在学习过程中，人类广泛的情感，如困惑、沮丧、无聊、焦虑、参与、惊讶和喜悦，会有规律地出现。不同的情绪最好用不同的方式来回应。例如，基于任务的反馈和指导是对困惑和沮丧情绪的有益反应，而同理心反馈更有助于愤怒或兴奋的情绪。先前的研究还没有回答情感适应如何在学生与基于计算机的互动学习环境中互动时最大限度地为他们带来好处的问题。然而，这个项目的研究人员现在已经做好了准备，可以很好地解决一个核心的、尚未回答的问题：学习环境如何适应学生的情感，创造最有效、最吸引人的学习体验，同时促进学习态度的改善。该项目将通过产生跨多个学科的理论和实践进步，提供重要的社会效益。该项目将导致对富有情感的学习的更深入的理解；一套广泛适用的情感适应原则；以及将纳入科学学习学习环境的情感适应和对话的计算模型。由此产生的情感建模技术可以作为下一代适应性教育软件的基础，该软件将通过丰富的情感适应来促进学习。这将在整个教育中广泛有用。该项目将通过与高度多样化的邓恩中学和哈内特中心中学合作，以及通过与STAR联盟的持续合作，扩大对计算的参与，来解决多样性问题。为了确保社会影响，研究结果将通过中学外展计划向公众传播，并通过在科学场所发表文章向科学界传播。该项目的三个主要科学目标是：(1)通过口头对话与完全工具化的学习环境互动，同时捕获丰富的多模式学生情感体验数据。观察性研究将通过让中学生与现有的科学教育学习环境互动来进行，该环境被称为“水晶岛”。水晶岛是由该项目的研究人员开发的，已经被数千名中学生在课堂上用来学习微生物学，但目前它不支持丰富的多模式互动或自然语言对话。水晶岛将全面收集丰富的多模式数据，包括语音、面部表情、凝视、姿势、皮肤电导反应、心率和解决问题的行动。(2)设计、开发和提炼情感理解模型，将学生的自然语言、非语言行为、生理反应和任务-行动现象整合到丰富的多维情感数据流中。通过利用从观察性研究中收集的这些数据，将使用包括隐马尔可夫建模在内的机器学习来构建情感理解模型。这将是第一个用于学习环境的情感-理解模型，它集成了口语(包括韵律、句法和语义)、非语言行为(包括凝视和姿势)、生理数据(包括皮肤电导反应和心率)和任务动作(包括学习环境中的导航和操作动作)的完整补充。(3)设计、开发和完善一个整合的情感和对话管理模型，自适应地响应学生在学习互动过程中的情感状态。通过利用观察性研究中收集的学习交互数据，通过强化学习、整合情感和对话管理，获得部分可观测的马尔可夫决策过程(POMDP)情感适应策略。由此产生的适应政策将管理系统在学生解决问题时对他们的情感做出反应的时间和方式。基于计算机的导师将提供解决问题的建议、鼓励、同理心反应和其他必要的支持，以改善教育经验和结果。

Affect, or emotion, profoundly shapes human experience. It influences how we perform tasks, how we build relationships with one another, and how we navigate the complexities of our daily lives. Affect is shaped and influenced by communication with other humans, experiences with the natural world, and interactions with machines. Affect plays a particularly prominent role in learning. During learning, a recurring subset of the broad range of human emotions such as confusion, frustration, boredom, anxiety, engagement, surprise, and delight appear regularly. Different emotions are best responded to in different ways. For example, task-based feedback and guidance is a helpful response to emotions of confusion and frustration, while empathetic feedback is more helpful for emotions of anger or excitement. Prior research has not answered the question of how affective adaptation can maximize the benefit to students as they interact with interactive computer-based learning environments. And yet the investigators on this project are now well positioned to address a central, unanswered question of how learning environments can adaptively respond to students' affect to create the most effective, engaging learning experiences while simultaneously promoting improved attitudes toward learning.The project will provide important societal benefits by generating theoretical and practical advances across multiple disciplines. The project will lead to a deeper understanding of affect-rich learning; a set of broadly applicable affect adaptation principles; and a computational model of affective adaptation and dialogue that will be incorporated into a learning environment for science learning. The resulting affect-modeling technologies can serve as a foundation for the next generation of adaptive educational software that will promote learning through affect-rich adaptation. This will be broadly useful throughout education. The project will address issues of diversity by partnering with the highly diverse Dunn Middle School and Harnett Central Middle School, and through ongoing collaboration with the STARS Alliance for Broadening Participation in Computing. To ensure societal impact, the results will be disseminated to the public through middle school outreach programs, and to the scientific community through publication at scientific venues.The three major scientific goals of the project are to: (1) Capture rich multimodal data of students' affective experiences while interacting with a fully instrumented learning environment with spoken dialogue. Observational studies will be conducted by having middle school students interact with an existing learning environment for science education called "Crystal Island." Crystal Island was developed by the investigators on this project and has already been used by thousands of students in middle school classrooms to learn microbiology, but it does not currently support rich multimodal interaction or natural language dialogue. Crystal Island will be fully instrumented to collect rich, multimodal data including speech, facial expression, gaze, posture, skin conductance response, heart rate, and problem-solving actions. (2) Design, develop, and refine an affect-understanding model that integrates students' natural language, nonverbal behavior, physiological response, and task-action phenomena into a rich multi-dimensional stream of affective data. By utilizing this data collected from the observational studies, an affect-understanding model will be constructed using machine learned including hidden Markov modeling. This will be the first affect-understanding model for learning environments that integrates the full complement of affect signals of spoken language (including prosody, syntax, and semantics), nonverbal behavior (including gaze and posture), physiological data (including skin conductance response and heart rate), and task actions (including navigation and manipulation actions in the learning environment). (3) Design, develop, and refine an integrated affect and dialogue management model that adaptively responds to students' affective states in the course of their learning interactions. By utilizing the learning-interaction data collected in the observational studies, a Partially Observable Markov Decision Process (POMDP) affect adaptation policy will be acquired with reinforcement learning, integrating affect and dialogue management. The resulting adaptation policy will govern both when and how the system responds to students' affect as they solve problems. The computer-based mentor will provide problem-solving advice, encouragement, empathetic responses, and other support as is needed to improve the educational experience and outcome.