Collaborative Research: CSL-MultiAD: Assessing Collaborative STEM Learning through Rich Information Flow based on Multi-Sensor Audio Diarization

学习概念的能力，特别是以科学和数学（STEM）为基础的学科，受到教育工作者的影响，他们激励、激励和创造支持性环境和教学方法，从而降低学生学习STEM学科的入门门槛。随着基于STEM的科学教育内容的扩展，以及基于课堂上先前科学接触的学生多样性的增加，全国范围内的教学资源一直受到限制。学生学习的一个关键方面是评估学生与学生之间以及教师与学生之间的人际交流的质量。在STEM学习中，能够提出正确问题的学生，知道他们理解什么以及他们需要帮助什么，使教育工作者能够构建他们的教学方法，帮助学生克服学习挑战。然而，到目前为止，在课堂上收集和测量学生与学生或学生与教师的语音通信实际上是不可能的。此外，目前的语音技术还不足以有效地克服课堂上的多说话者和自然交流。该项目将开发课堂音频收集和测量工具，以供学生共同解决问题，以及教师与学生个人/小组的参与。音频收集解决方案包括教室学生子集上的个人录音机，以及每个学生组内的中央智能扬声器麦克风收集单元。计算机程序将被开发来分析谁在说话，什么时候说话，以及发现对STEM主题和学习评估感兴趣的关键词。隐私得以维护，因为音频分析的重点是高层次的措施，如学生个人字数统计、每个说话者的匿名标记，以及学生和教师之间的对话转换。教师驱动的关键字集将用于帮助衡量哪些学生在理解概念方面有问题。这些单独的交流测量术语将被整合到仪表板显示中，使教师能够轻松使用学生参与STEM学习的反馈。该项目有可能通过课堂交流提高评估学习的能力，并有可能帮助教师更有效地指导他们的时间/专业知识，以改善学生的STEM学习。该项目将通过测量学生与同龄人之间以及教师与学生之间的人际交流参与的质量，开发评估课堂学习的方法。研究表明，如果学生与学生、学生与教师在语音交流中进行动态互动，学习效果会得到改善。该项目在教室里引入了个人录音机，以记录一整天的语音互动。接下来，这些多麦克风录音流被汇集在一起，语音和语言处理算法将被制定以执行“音频diarization”——确定“谁说了什么，什么时候说”的过程，并根据课堂主题确定潜在的兴趣关键词。数字化的输出将推动度量标准的制定，以评估沟通参与。从单个音频流（字数统计、通话时间、轮次、关键字配置文件）中衍生的基于通信的特征将通过音频化以每个学生为基础进行提取。接下来，这个信息流将用于开发基于类的群体动力学。该解决方案为教师提供了一种监测学生在科学活动领域长期参与的方法，帮助教师识别没有口头参与科学话语的学生，并快速评估课堂实践变化对改善学习的影响。使用语音活动检测、基于机器学习模型的说话人分类以及用于科学主题识别和跟踪的关键字识别，将解决基于自然音频数据的自动音频流语音处理的许多技术挑战。这些研究目标将在课堂环境中进行评估，并得到教师对最终解决方案有效性的反馈。由此产生的语音技术进步将为未来的智能教室提供新的机会，教师可以通过语音评估更好地评估学生对科学的参与程度，而不是不常见的传统标准化测试。最终，这一努力将为教师提供工具，以识别和经常监测脱离科学学习的早期指标，并有可能提高代表性不足的学生群体对科学的兴趣，并进一步使STEM劳动力多样化。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The ability to learn concepts, especially for science and math (STEM) based disciplines, is impacted by educators who inspire, motivate, and create supportive environments and teaching methodologies which lower the entry barrier for students learning STEM subjects. Teaching resources nationwide have historically been constrained as STEM based science content for education expands with increasing student diversity based on prior science exposure in the classroom. A key aspect of student learning is to assess the quality of human communications between student-and-student as well as teacher-and-student. In STEM learning, students who are able to ask the right questions, know what they understand as well as what they need help with, allows educators to structure their teaching methods to help students overcome learning challenges. However, to date, it has been virtually impossible to collect and measure student-to-student or student-to-teacher voice communications in the classroom. Also, current speech technology is not sufficiently effective to overcome multi-speaker and naturalistic communications in classrooms. This project will develop classroom audio collection and measurement tools for students working together to solve problems, as well as teacher involvement with individual/groups of students. The audio collection solution includes both individual recorders on a sub-set of classroom students, as well as central smart speaker microphone collection units within each student group. Computer programs will be developed to analyze who is speaking and when, as well as spot keywords of interest for STEM topics and learning assessment. Privacy is maintained, since audio analysis is focused on high level measures such as individual student word counts, anonymous tagging of each speaker, and connecting conversational turns between students and teachers. A teacher driven keyword set will be used to help measure which students are having problems understanding concepts. These individual communication measured terms will be integrated into a dashboard display, to empower teachers with easy to use feedback on student engagement for STEM learning. The project has the potential to improve the ability to assess learning through classroom communications, and potentially help teachers better direct their time/expertise more efficiently to improve STEM learning for students. This project will develop ways to assess learning in classrooms by measuring the quality of human communication engagement between students-and-peers as well as teachers-and-students. Research has shown that learning is improved if there is dynamic interaction between student-to-student and student-to-teacher in voice communications. The project introduces personal recorders in the classroom to capture voice interactions during the entire day. Next, these multi-microphone recording streams are pooled together, where speech and language processing algorithms will be formulated to perform "audio diarization" - the process of determining "who spoke, what, and when", with potential keywords of interest based on classroom topics identified. The diarization output will drive the formulation of metrics to assess communication engagement. Communication based features derived from individual audio streams (word count, talk time, turn-taking, keyword profile) will be extracted on a per student basis through audio diarization. Next, this information flow will be used to develop class based group dynamics. This solution represents an approach for teachers to monitor student engagement over time in science activity areas, helping teachers identify students who are not verbally engaged in science discourse and quickly assess the impact of changes in classroom practices to improve learning. A number of technology challenges will be addressed for automatic audio stream based voice processing of naturalistic audio data using speech activity detection, speaker diarization based on machine learning models, and keyword spotting for science topic identification and tracking. These research aims will be assessed in classroom settings with teacher feedback on the effectiveness of the resulting solutions. The resulting speech technology advancements would offer new opportunities for future smart classrooms for voice assessment for teachers to better assess student involvement in science vs. infrequent traditional standardized testing. Ultimately, this effort will equip teachers with tools to identify and frequently monitor early indicators of disengagement in science learning, and potentially increase science interest by under-represented student populations and further diversify the STEM workforce.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.