CHS: Medium: Adapting to Affect in Multimodal Dialogue-Rich Interaction with Middle School Students

情感或情绪深刻地塑造了人类的经验。它影响着我们如何执行任务，如何与他人建立关系，以及如何驾驭日常生活的复杂性。情感是通过与其他人的交流、与自然世界的体验以及与机器的互动来塑造和影响的。情感在学习中起着特别重要的作用。在学习过程中，困惑、沮丧、无聊、焦虑、投入、惊讶和喜悦等人类情感的一个循环子集会定期出现。不同的情绪最好以不同的方式回应。例如，基于任务的反馈和指导是对困惑和沮丧情绪的有益反应，而移情反馈对愤怒或兴奋情绪更有帮助。先前的研究还没有回答的问题，如何情感适应可以最大限度地提高学生的利益，因为他们与互动的计算机为基础的学习环境。然而，这个项目的研究人员现在已经做好了充分的准备，以解决一个核心的，没有答案的问题，即学习环境如何适应性地回应学生的影响，创造最有效的，最吸引人的学习经验，同时促进改善学习态度。该项目将通过产生跨多学科的理论和实践进步，提供重要的社会效益。该项目将导致情感丰富的学习更深入的理解;一套广泛适用的影响适应原则;和情感适应和对话的计算模型，将被纳入科学学习的学习环境。由此产生的情感建模技术可以作为下一代自适应教育软件的基础，通过丰富的情感适应来促进学习。这将在整个教育过程中广泛发挥作用。该项目将通过与高度多样化的邓恩中学和哈内特中央中学合作，并通过与扩大参与计算的STARS联盟持续合作，解决多样性问题。为了确保社会影响，研究结果将通过中学外展计划向公众传播，并通过在科学场所发表向科学界传播。该项目的三个主要科学目标是：（1）在与完全仪器化的学习环境进行交互时，通过口语对话捕获学生情感体验的丰富多模态数据。通过让中学生与一个名为"水晶岛"的现有科学教育学习环境互动，将进行观察性研究。“水晶岛是由该项目的研究人员开发的，已经被数千名学生在中学课堂上用于学习微生物学，但它目前不支持丰富的多模式交互或自然语言对话。水晶岛将配备完整的仪器来收集丰富的多模式数据，包括言语、面部表情、凝视、姿势、皮肤电导反应、心率和解决问题的行动。(2)设计，开发和完善情感理解模型，将学生的自然语言，非语言行为，生理反应和任务动作现象整合到丰富的多维情感数据流中。通过利用从观察性研究中收集的这些数据，将使用机器学习（包括隐马尔可夫模型）构建情感理解模型。这将是第一个用于学习环境的情感理解模型，它集成了口语（包括韵律，句法和语义），非语言行为（包括凝视和姿势），生理数据（包括皮肤电导反应和心率）和任务动作（包括学习环境中的导航和操作动作）的全部情感信号。(3)设计，开发和完善一个综合的情感和对话管理模型，自适应地响应学生的情感状态，在他们的学习互动过程中。通过利用在观察性研究中收集的学习交互数据，部分可观察马尔可夫决策过程（POMDP）的情感适应策略将与强化学习，整合情感和对话管理。由此产生的适应政策将决定系统何时以及如何在学生解决问题时对他们的情感做出反应。基于计算机的导师将提供解决问题的建议，鼓励，同情的反应，以及其他需要改善教育经验和结果的支持。

Affect, or emotion, profoundly shapes human experience. It influences how we perform tasks, how we build relationships with one another, and how we navigate the complexities of our daily lives. Affect is shaped and influenced by communication with other humans, experiences with the natural world, and interactions with machines. Affect plays a particularly prominent role in learning. During learning, a recurring subset of the broad range of human emotions such as confusion, frustration, boredom, anxiety, engagement, surprise, and delight appear regularly. Different emotions are best responded to in different ways. For example, task-based feedback and guidance is a helpful response to emotions of confusion and frustration, while empathetic feedback is more helpful for emotions of anger or excitement. Prior research has not answered the question of how affective adaptation can maximize the benefit to students as they interact with interactive computer-based learning environments. And yet the investigators on this project are now well positioned to address a central, unanswered question of how learning environments can adaptively respond to students' affect to create the most effective, engaging learning experiences while simultaneously promoting improved attitudes toward learning.The project will provide important societal benefits by generating theoretical and practical advances across multiple disciplines. The project will lead to a deeper understanding of affect-rich learning; a set of broadly applicable affect adaptation principles; and a computational model of affective adaptation and dialogue that will be incorporated into a learning environment for science learning. The resulting affect-modeling technologies can serve as a foundation for the next generation of adaptive educational software that will promote learning through affect-rich adaptation. This will be broadly useful throughout education. The project will address issues of diversity by partnering with the highly diverse Dunn Middle School and Harnett Central Middle School, and through ongoing collaboration with the STARS Alliance for Broadening Participation in Computing. To ensure societal impact, the results will be disseminated to the public through middle school outreach programs, and to the scientific community through publication at scientific venues.The three major scientific goals of the project are to: (1) Capture rich multimodal data of students' affective experiences while interacting with a fully instrumented learning environment with spoken dialogue. Observational studies will be conducted by having middle school students interact with an existing learning environment for science education called "Crystal Island." Crystal Island was developed by the investigators on this project and has already been used by thousands of students in middle school classrooms to learn microbiology, but it does not currently support rich multimodal interaction or natural language dialogue. Crystal Island will be fully instrumented to collect rich, multimodal data including speech, facial expression, gaze, posture, skin conductance response, heart rate, and problem-solving actions. (2) Design, develop, and refine an affect-understanding model that integrates students' natural language, nonverbal behavior, physiological response, and task-action phenomena into a rich multi-dimensional stream of affective data. By utilizing this data collected from the observational studies, an affect-understanding model will be constructed using machine learned including hidden Markov modeling. This will be the first affect-understanding model for learning environments that integrates the full complement of affect signals of spoken language (including prosody, syntax, and semantics), nonverbal behavior (including gaze and posture), physiological data (including skin conductance response and heart rate), and task actions (including navigation and manipulation actions in the learning environment). (3) Design, develop, and refine an integrated affect and dialogue management model that adaptively responds to students' affective states in the course of their learning interactions. By utilizing the learning-interaction data collected in the observational studies, a Partially Observable Markov Decision Process (POMDP) affect adaptation policy will be acquired with reinforcement learning, integrating affect and dialogue management. The resulting adaptation policy will govern both when and how the system responds to students' affect as they solve problems. The computer-based mentor will provide problem-solving advice, encouragement, empathetic responses, and other support as is needed to improve the educational experience and outcome.