Using Neural Networks for Automated Classification of Elementary Mathematics Instructional Activities

该研究项目得到了EHR核心研究（ECR）计划的支持，该计划支持推进STEM学习和学习环境基础研究、扩大STEM参与和STEM劳动力发展的工作。在过去的十年里，使用视频来为教师做准备和研究教学质量有了巨大的增长。对教学候选人和初任教师进行总结性评估的突出方法是录制教学实践的视频。此外，大规模的研究还包括教学视频。教师预备课程越来越多地在方法课程和形成性评估中使用视频。此外，在职前准备和在职专业发展中越来越多地使用交互式模拟，依赖于录像的示范教学实践。尽管近年来使用视频来衡量和提高教学质量有了显著增长，但大规模使用它仍然存在一些关键挑战。首先，对于训练有素的人类评分员来说，观看数百小时的视频课程是非常耗时的。其次，每个视频使用多个人工评分员会增加财务成本和时间需求。第三，手动编目、标记和索引大量的课堂视频以供以后观看是非常耗时的。计算机视觉、机器学习和深度学习的最新进展可能为这些挑战提供解决方案，并可能使分析和评分视频的过程更有效。特别是，深度学习已经成为分析视频内容相关问题的最先进选择。这项拟议的NSF核心研究将利用为NSF资助的两项研究收集的小学数学教学视频，这些研究以数学扫描（M-Scan）课堂观察工具为特色。本研究将利用这些视频探索几种方法，利用深度神经网络对数学教学视频中的教学活动进行分类。本研究将通过检验三种类型的人工神经网络在多大程度上能够准确地对小学数学教学视频中的(a)对象和(b)教学活动进行分类，从而促进知识和理解。例如，神经网络可以很直接地确定小学教师是在讲课还是在促进与学生的讨论。另一方面，这样的网络可能更难评估教师表达数学内容的方式、问题的性质，以及学生的手势是否意味着理解。研究设计涉及使用计算机视觉、机器学习和深度学习的三个方面：(a)神经网络的类型，(b)视频标签的类型（即对象标签和教学活动标签），以及(c)数学教学的主题（例如，数字和运算；模式、函数和代数；几何）。最近有几种类型的神经网络被证明对视频分类是有效的：卷积神经网络（cnn），长短期记忆（LSTM）神经网络，以及cnn -LSTM混合神经网络。最终，这个项目的目标是开始建立系统化的基础设施，以高效和经济的方式对课堂教学视频进行大规模分类。研究结果可能会对以下方面产生关键影响：(a)以教学视频为特征的大规模研究；(b)职前教师准备计划，在职专业发展活动，以及评估教学候选人和实习教师的努力。特别地，本研究的结果将告知有关可用于正确分类教学视频的神经网络类型的决策，以及为此目的使用网络的实际限制。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This research project is supported by the EHR Core Research (ECR) program, which supports work that advances fundamental research on STEM learning and learning environments, broadening participation in STEM, and STEM workforce development.In the last decade, there has been a tremendous increase in the use of video for preparing teachers and studying teaching quality. Prominent approaches to summative evaluation of teaching candidates and beginning teachers feature video recordings of instructional practices. In addition, large-scale research studies have included video of instruction. Teacher preparation programs increasingly use video in methods courses and for formative assessment purposes. Moreover, the growing use of interactive simulation in pre-service preparation and in-service professional development relies on having video recorded examples of exemplary instructional practices. Despite the significant growth in the use of video to measure and promote instructional quality in recent years, there remain some key challenges to employing it at scale. First, it is very time-consuming for trained human raters to view hundreds of hours of video recorded lessons. Second, financial costs and time demands increase with the use of multiple human raters per video. Third, manually cataloging, labeling, and indexing large volumes of classroom video for later viewing is time consuming. Recent advances in computer vision, machine learning, and deep learning may provide solutions to these challenges and could make the process of analyzing and scoring videos more efficient. In particular, deep learning has become the state-of-the-art choice in problems related to analyzing the content of video. This proposed NSF Core Research study will draw on videos of elementary mathematics instruction that were collected for two NSF-funded studies that featured the Mathematics-Scan (M-Scan) classroom observation tool. The research will use these videos to explore several ways that deep neural networks can be used to classify instructional activities in videos of math instruction.This study will advance knowledge and understanding by examining the degree to which three types of artificial neural networks can accurately classify (a) objects and (b) instructional activities in videos of elementary mathematics instruction. For example, it may be straightforward for neural networks to determine whether an elementary teacher is engaged in lecture vs. facilitating discussion with students. On the other hand, it may be harder for such networks to assess the ways teachers represent math content, the nature of their questions, and whether student gestures signify understanding. The research design concerns three aspects of using computer vision, machine learning, and deep learning: (a) the type of neural network, (b) the type of video label (i.e., object labels and instructional activity labels), and (c) the subject of math instruction (e.g., number and operations; patterns, functions, and algebra; geometry). A few types of neural networks have recently proven effective for video classification: convolutional neural networks (CNNs), long short-term memory (LSTM) neural networks, and hybrid CNN-LSTMs. Ultimately, this project is aimed at beginning to build systematic infrastructure for classifying videos of classroom instruction at scale in efficient and affordable ways. The findings will potentially have key implications for (a) large-scale research studies that feature videos of instruction and (b) pre-service teacher preparation programs, in-service professional development activities, and efforts to evaluate teaching candidates and practicing teachers. In particular, the results from this study will inform decisions about the types of neural networks that can be used to correctly classify videos of instruction and the practical limitations of using networks for this purpose.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.