Collaborative Research: CSL-MultiAD: Assessing Collaborative STEM Learning through Rich Information Flow based on Multi-Sensor Audio Diarization

点击翻译按钮获取中文摘要

The ability to learn concepts, especially for science and math (STEM) based disciplines, is impacted by educators who inspire, motivate, and create supportive environments and teaching methodologies which lower the entry barrier for students learning STEM subjects. Teaching resources nationwide have historically been constrained as STEM based science content for education expands with increasing student diversity based on prior science exposure in the classroom. A key aspect of student learning is to assess the quality of human communications between student-and-student as well as teacher-and-student. In STEM learning, students who are able to ask the right questions, know what they understand as well as what they need help with, allows educators to structure their teaching methods to help students overcome learning challenges. However, to date, it has been virtually impossible to collect and measure student-to-student or student-to-teacher voice communications in the classroom. Also, current speech technology is not sufficiently effective to overcome multi-speaker and naturalistic communications in classrooms. This project will develop classroom audio collection and measurement tools for students working together to solve problems, as well as teacher involvement with individual/groups of students. The audio collection solution includes both individual recorders on a sub-set of classroom students, as well as central smart speaker microphone collection units within each student group. Computer programs will be developed to analyze who is speaking and when, as well as spot keywords of interest for STEM topics and learning assessment. Privacy is maintained, since audio analysis is focused on high level measures such as individual student word counts, anonymous tagging of each speaker, and connecting conversational turns between students and teachers. A teacher driven keyword set will be used to help measure which students are having problems understanding concepts. These individual communication measured terms will be integrated into a dashboard display, to empower teachers with easy to use feedback on student engagement for STEM learning. The project has the potential to improve the ability to assess learning through classroom communications, and potentially help teachers better direct their time/expertise more efficiently to improve STEM learning for students. This project will develop ways to assess learning in classrooms by measuring the quality of human communication engagement between students-and-peers as well as teachers-and-students. Research has shown that learning is improved if there is dynamic interaction between student-to-student and student-to-teacher in voice communications. The project introduces personal recorders in the classroom to capture voice interactions during the entire day. Next, these multi-microphone recording streams are pooled together, where speech and language processing algorithms will be formulated to perform "audio diarization" - the process of determining "who spoke, what, and when", with potential keywords of interest based on classroom topics identified. The diarization output will drive the formulation of metrics to assess communication engagement. Communication based features derived from individual audio streams (word count, talk time, turn-taking, keyword profile) will be extracted on a per student basis through audio diarization. Next, this information flow will be used to develop class based group dynamics. This solution represents an approach for teachers to monitor student engagement over time in science activity areas, helping teachers identify students who are not verbally engaged in science discourse and quickly assess the impact of changes in classroom practices to improve learning. A number of technology challenges will be addressed for automatic audio stream based voice processing of naturalistic audio data using speech activity detection, speaker diarization based on machine learning models, and keyword spotting for science topic identification and tracking. These research aims will be assessed in classroom settings with teacher feedback on the effectiveness of the resulting solutions. The resulting speech technology advancements would offer new opportunities for future smart classrooms for voice assessment for teachers to better assess student involvement in science vs. infrequent traditional standardized testing. Ultimately, this effort will equip teachers with tools to identify and frequently monitor early indicators of disengagement in science learning, and potentially increase science interest by under-represented student populations and further diversify the STEM workforce.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.