CAREER: Developing New Scientific Instruments for Classroom Observation: A Multi-modal Machine Learning Approach

该项目将利用人工智能(AI)来提高课堂教学质量和教育研究的精确度，为教师和科学家提供观察教师、学生和他们的同龄人之间的人际动态的新方法。几十年的研究表明，教师和学生之间互动的质量和数量可以对学生的参与度、学习态度以及下游的学业和社会情感结果产生重大影响。尽管在研究课堂互动及其对学生学习的影响方面取得了进展，并在制定有效干预措施帮助教师更好地教学方面取得了进展，但教育衡量的现状目前是教育研究和教师培训进一步取得进展的一个重要障碍。当代方法的具体问题包括忽视个别学生和少数群体可能存在的不同课堂体验，以及仅为教师提供有限的可操作反馈。这个项目将偏离标准的观察方案，这些方案通常描述“普通”学习者的“平均”课堂体验，而是专注于随着时间的推移表征教室中每个学生的细粒度体验。在这个项目中开发的科学仪器也将被用来帮助教师在与课堂上的特定学生互动时识别潜在的偏见。为了实现这些目标，该团队将在多模式(视觉、语音、自然语言)机器学习方面取得进展，以设计新的架构来分析学校教室的视频并感知细粒度的互动。设想的人工智能系统将(1)识别谁在学校课堂上与谁互动，何时以何种方式互动；(2)在教学过程中找到对教师反馈最重要的关键事件；以及(3)从不同的维度总结每个学生的互动，以找到需要更多关注的学生，并揭示可能的偏见。基于这些新的分析，该团队将开发(4)预测模型来估计社会情感和学术结果。最后，该团队将(5)设计新的教师培训经验，帮助教师更准确地感知课堂动态。该项目将对计算机视觉、语音分析、机器学习和教育领域做出贡献，并将为自动说话人二元化、人物跟踪、情绪分析和课堂观察分析提供新的见解。科学和教育议程为研究助理提供了跨学科培训的机会；它们还将促进研究团队与马萨诸塞州和弗吉尼亚州教师之间的合作并从中受益。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project will harness artificial intelligence (AI) to improve both the quality of classroom teaching and the precision of educational research by providing teachers and scientists with new methods of observing the inter-personal dynamics between teachers, students, and their peers. Decades of research have demonstrated that the quality and quantity of interactions between teachers and students can have a major impact on student engagement, attitudes toward learning, and downstream academic and socio-emotional outcomes. Despite the progress that has been made in studying classroom interactions and their impact on students' learning, as well as in developing effective interventions to help teachers teach better, the status quo of educational measurement is currently a significant roadblock to further progress in both educational research and teacher training. Specific problems with contemporary methods include ignoring the possibly different classroom experiences of individual students and minority subgroups, and providing only limited actionable feedback for teachers. This project will depart from standard observation protocols, which typically describe the "average" classroom experience of the "average" learner, and instead focus on characterizing over time the fine-grained experiences of every student in the classroom. The scientific instruments developed during this project will also be used to help teachers to identify potential biases when interacting with particular students in their classes.To achieve these goals, the team will make advances in multi-modal (vision, speech, natural language) machine learning to devise new architectures that analyze videos of school classrooms and perceive fine-grained interactions. The envisioned AI systems will (1) identify who is interacting with whom, when, and how in a school classroom; (2) find the key events during a teaching session that are most important for teacher feedback; and (3) summarize interactions for each student along different dimensions to find students who need more attention and to uncover possible bias. Based on these new analyses, the team will develop (4) predictive models to estimate socioemotional and academic outcomes outcomes. Finally, the team will (5) devise new teacher training experiences that help teachers to perceive classroom dynamics more accurately. The project will result in contributions to the fields of computer vision, speech analysis, machine learning, and education, and will offer new insights into automatic speaker diarization, person tracking, sentiment analysis, and classroom observation analysis. The scientific and educational agendas provide opportunities for inter-disciplinary training of research assistants; they will also enable and benefit from collaboration between the research team and teachers in both Massachusetts and Virginia.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.