CHS: Medium: Adapting to Affect in Multimodal Dialogue-Rich Interaction with Middle School Students

情感或情感深刻地塑造了人类的经历。它影响着我们如何完成任务，我们如何与他人建立关系，以及我们如何应对日常生活的复杂性。情感是由与他人的交流、与自然世界的经历以及与机器的互动形成和影响的。情感在学习中起着特别突出的作用。在学习过程中，人类情感的一个反复出现的子集，如困惑、沮丧、无聊、焦虑、投入、惊讶和喜悦，经常出现。不同的情绪有不同的应对方式。例如，基于任务的反馈和指导是对困惑和沮丧情绪的有益反应，而移情反馈对愤怒或兴奋情绪更有帮助。先前的研究没有回答情感适应如何在学生与交互式计算机学习环境互动时最大化学生的利益。然而，这个项目的研究人员现在很好地定位于解决一个核心问题，即学习环境如何能够自适应地响应学生的影响，以创造最有效、最吸引人的学习体验，同时促进改善学习态度。该项目将通过产生跨多个学科的理论和实践进步，提供重要的社会效益。该项目将导致对富情感学习的更深入理解；一套广泛适用的适应影响原则；情感适应和对话的计算模型将被纳入科学学习的学习环境。由此产生的情感建模技术可以作为下一代适应性教育软件的基础，这些软件将通过富有情感的适应性来促进学习。这将在整个教育过程中广泛有用。该项目将通过与高度多样化的邓恩中学和哈尼特中央中学合作，并通过与STARS联盟的持续合作来解决多样性问题，以扩大计算机的参与。为了确保社会影响，研究结果将通过中学外展项目向公众传播，并通过在科学场所出版向科学界传播。该项目的三个主要科学目标是：(1)获取学生情感体验的丰富多模态数据，同时与具有口语对话的全仪器学习环境进行交互。观察性研究将让中学生与现有的科学教育学习环境“水晶岛”进行互动。Crystal Island是由该项目的研究人员开发的，已经被成千上万的中学生用于学习微生物学，但它目前不支持丰富的多模式交互或自然语言对话。水晶岛将配备齐全的设备，收集丰富的多模式数据，包括语音、面部表情、凝视、姿势、皮肤电导反应、心率和解决问题的行动。(2)设计、开发并完善情感理解模型，将学生的自然语言、非语言行为、生理反应和任务-动作现象整合为丰富的多维情感数据流。通过利用从观察研究中收集的数据，将使用包含隐马尔可夫建模的机器学习构建情感理解模型。这将是第一个学习环境的情感理解模型，它整合了口语（包括韵律、句法和语义）、非语言行为（包括凝视和姿势）、生理数据（包括皮肤电导反应和心率）和任务动作（包括学习环境中的导航和操作动作）的情感信号的全部内容。(3)设计、开发并完善情感与对话一体化管理模型，该模型能够自适应地响应学生在学习互动过程中的情感状态。利用观察性研究中收集的学习交互数据，通过强化学习，整合情感和对话管理，获得部分可观察马尔可夫决策过程（POMDP）影响适应策略。由此产生的适应政策将管理系统何时以及如何响应学生在解决问题时的影响。计算机导师将提供解决问题的建议、鼓励、移情反应和其他必要的支持，以改善教育体验和成果。

Affect, or emotion, profoundly shapes human experience. It influences how we perform tasks, how we build relationships with one another, and how we navigate the complexities of our daily lives. Affect is shaped and influenced by communication with other humans, experiences with the natural world, and interactions with machines. Affect plays a particularly prominent role in learning. During learning, a recurring subset of the broad range of human emotions such as confusion, frustration, boredom, anxiety, engagement, surprise, and delight appear regularly. Different emotions are best responded to in different ways. For example, task-based feedback and guidance is a helpful response to emotions of confusion and frustration, while empathetic feedback is more helpful for emotions of anger or excitement. Prior research has not answered the question of how affective adaptation can maximize the benefit to students as they interact with interactive computer-based learning environments. And yet the investigators on this project are now well positioned to address a central, unanswered question of how learning environments can adaptively respond to students' affect to create the most effective, engaging learning experiences while simultaneously promoting improved attitudes toward learning.The project will provide important societal benefits by generating theoretical and practical advances across multiple disciplines. The project will lead to a deeper understanding of affect-rich learning; a set of broadly applicable affect adaptation principles; and a computational model of affective adaptation and dialogue that will be incorporated into a learning environment for science learning. The resulting affect-modeling technologies can serve as a foundation for the next generation of adaptive educational software that will promote learning through affect-rich adaptation. This will be broadly useful throughout education. The project will address issues of diversity by partnering with the highly diverse Dunn Middle School and Harnett Central Middle School, and through ongoing collaboration with the STARS Alliance for Broadening Participation in Computing. To ensure societal impact, the results will be disseminated to the public through middle school outreach programs, and to the scientific community through publication at scientific venues.The three major scientific goals of the project are to: (1) Capture rich multimodal data of students' affective experiences while interacting with a fully instrumented learning environment with spoken dialogue. Observational studies will be conducted by having middle school students interact with an existing learning environment for science education called "Crystal Island." Crystal Island was developed by the investigators on this project and has already been used by thousands of students in middle school classrooms to learn microbiology, but it does not currently support rich multimodal interaction or natural language dialogue. Crystal Island will be fully instrumented to collect rich, multimodal data including speech, facial expression, gaze, posture, skin conductance response, heart rate, and problem-solving actions. (2) Design, develop, and refine an affect-understanding model that integrates students' natural language, nonverbal behavior, physiological response, and task-action phenomena into a rich multi-dimensional stream of affective data. By utilizing this data collected from the observational studies, an affect-understanding model will be constructed using machine learned including hidden Markov modeling. This will be the first affect-understanding model for learning environments that integrates the full complement of affect signals of spoken language (including prosody, syntax, and semantics), nonverbal behavior (including gaze and posture), physiological data (including skin conductance response and heart rate), and task actions (including navigation and manipulation actions in the learning environment). (3) Design, develop, and refine an integrated affect and dialogue management model that adaptively responds to students' affective states in the course of their learning interactions. By utilizing the learning-interaction data collected in the observational studies, a Partially Observable Markov Decision Process (POMDP) affect adaptation policy will be acquired with reinforcement learning, integrating affect and dialogue management. The resulting adaptation policy will govern both when and how the system responds to students' affect as they solve problems. The computer-based mentor will provide problem-solving advice, encouragement, empathetic responses, and other support as is needed to improve the educational experience and outcome.