CAREER: Developing New Scientific Instruments for Classroom Observation: A Multi-modal Machine Learning Approach

该项目将利用人工智能（AI）提高课堂教学质量和教育研究的准确性，为教师和科学家提供观察教师、学生和同龄人之间人际关系动态的新方法。几十年的研究表明，师生互动的质量和数量会对学生的参与度、学习态度以及下游的学术和社会情感结果产生重大影响。尽管在研究课堂互动及其对学生学习的影响，以及制定有效的干预措施以帮助教师更好地教学方面取得了进展，但教育测量的现状目前是教育研究和教师培训进一步取得进展的重大障碍。当代方法的具体问题包括忽视个别学生和少数群体可能不同的课堂体验，并仅为教师提供有限的可操作反馈。该项目将脱离标准观察协议，标准观察协议通常描述“普通”学习者的“平均”课堂体验，而是专注于描述课堂上每个学生的细粒度体验。在这个项目中开发的科学仪器也将用于帮助教师在与课堂上的特定学生互动时识别潜在的偏见。为了实现这些目标，该团队将在多模态（视觉、语音、自然语言）机器学习方面取得进展，以设计新的架构，分析学校教室的视频并感知细粒度的交互。设想中的人工智能系统将(1)识别谁在学校教室里与谁、何时以及如何互动；(2)找出教学过程中最重要的关键事件，以供教师反馈；(3)总结每个学生在不同维度上的互动，发现需要更多关注的学生，并发现可能存在的偏见。基于这些新的分析，该团队将开发(4)预测模型来估计社会情感和学术成果。最后，该团队将(5)设计新的教师培训体验，帮助教师更准确地感知课堂动态。该项目将为计算机视觉、语音分析、机器学习和教育领域做出贡献，并将为自动说话人化、人物跟踪、情感分析和课堂观察分析提供新的见解。科学和教育议程为研究助理的跨学科培训提供了机会；他们还将从马萨诸塞州和弗吉尼亚州的研究团队和教师之间的合作中受益。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project will harness artificial intelligence (AI) to improve both the quality of classroom teaching and the precision of educational research by providing teachers and scientists with new methods of observing the inter-personal dynamics between teachers, students, and their peers. Decades of research have demonstrated that the quality and quantity of interactions between teachers and students can have a major impact on student engagement, attitudes toward learning, and downstream academic and socio-emotional outcomes. Despite the progress that has been made in studying classroom interactions and their impact on students' learning, as well as in developing effective interventions to help teachers teach better, the status quo of educational measurement is currently a significant roadblock to further progress in both educational research and teacher training. Specific problems with contemporary methods include ignoring the possibly different classroom experiences of individual students and minority subgroups, and providing only limited actionable feedback for teachers. This project will depart from standard observation protocols, which typically describe the "average" classroom experience of the "average" learner, and instead focus on characterizing over time the fine-grained experiences of every student in the classroom. The scientific instruments developed during this project will also be used to help teachers to identify potential biases when interacting with particular students in their classes.To achieve these goals, the team will make advances in multi-modal (vision, speech, natural language) machine learning to devise new architectures that analyze videos of school classrooms and perceive fine-grained interactions. The envisioned AI systems will (1) identify who is interacting with whom, when, and how in a school classroom; (2) find the key events during a teaching session that are most important for teacher feedback; and (3) summarize interactions for each student along different dimensions to find students who need more attention and to uncover possible bias. Based on these new analyses, the team will develop (4) predictive models to estimate socioemotional and academic outcomes outcomes. Finally, the team will (5) devise new teacher training experiences that help teachers to perceive classroom dynamics more accurately. The project will result in contributions to the fields of computer vision, speech analysis, machine learning, and education, and will offer new insights into automatic speaker diarization, person tracking, sentiment analysis, and classroom observation analysis. The scientific and educational agendas provide opportunities for inter-disciplinary training of research assistants; they will also enable and benefit from collaboration between the research team and teachers in both Massachusetts and Virginia.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.