Collaborative Research: CSL-MultiAD: Assessing Collaborative STEM Learning through Rich Information Flow based on Multi-Sensor Audio Diarization

学习概念的能力，特别是基于科学和数学(STEM)学科的学习能力，受到教育工作者的影响，他们激励、激励和创造支持性的环境和教学方法，降低学生学习STEM学科的入学门槛。全国范围内的教学资源历来受到限制，因为基于STEM的教育科学内容随着学生多样性的增加而扩大，这是基于课堂上先前的科学接触。学生学习的一个关键方面是评估学生与学生之间以及教师与学生之间的人际交流的质量。在STEM学习中，能够提出正确的问题、了解他们理解的内容以及需要帮助的学生，使教育工作者能够构建他们的教学方法，帮助学生克服学习挑战。然而，到目前为止，在课堂上收集和测量学生对学生或学生对教师的语音通信几乎是不可能的。此外，目前的语音技术还不足以有效地克服课堂上的多发言者和自然主义交流。该项目将开发课堂音频收集和测量工具，供学生合作解决问题，以及教师与学生个人/团体的参与。音频采集解决方案既包括教室学生子集上的单个录音机，也包括每个学生组中的中央智能扬声器麦克风采集单元。将开发计算机程序来分析谁在何时发言，以及为STEM主题和学习评估寻找感兴趣的关键字。由于音频分析侧重于高水平的措施，如个别学生字数统计、每个说话者的匿名标记以及连接学生和教师之间的对话，因此隐私得到了保护。教师主导的关键词集将被用来帮助衡量哪些学生在理解概念方面存在问题。这些单独的交流测量术语将被集成到仪表板显示中，使教师能够轻松使用关于学生参与STEM学习的反馈。该项目有可能提高通过课堂交流评估学习的能力，并有可能帮助教师更有效地将他们的时间/专业知识用于改善学生的STEM学习。该项目将通过衡量学生与同龄人以及教师与学生之间的人际交流参与的质量，开发评估课堂学习的方法。研究表明，如果在语音交流中学生对学生和学生对教师之间存在动态互动，学习会得到改善。该项目在教室里引入了个人录音机，以捕捉全天的语音互动。下一步，这些多麦克风录音流被汇集在一起，其中语音和语言处理算法将被制定来执行“音频二值化”--即根据确定的课堂主题确定“谁发言、什么内容和何时发言”的过程，以及潜在的感兴趣的关键字。二元化输出将推动制定评估沟通参与度的指标。从单个音频流中提取的基于通信的特征(字数、通话时间、话轮转换、关键字配置文件)将通过音频二值化在每个学生的基础上提取。接下来，这个信息流将被用来开发基于类的组动态。这一解决方案为教师提供了一种监控学生随时间参与科学活动领域的方法，帮助教师识别没有口头参与科学话语的学生，并快速评估课堂实践变化对改善学习的影响。将解决基于自动音频流的自然音频数据的语音处理(使用语音活动检测)、基于机器学习模型的说话人二元化以及用于科学主题识别和跟踪的关键字识别的许多技术挑战。这些研究目标将在课堂环境中进行评估，教师将就由此产生的解决方案的有效性进行反馈。由此产生的语音技术进步将为未来用于语音评估的智能教室提供新的机会，使教师能够更好地评估学生对科学的参与程度，而不是不常见的传统标准化测试。最终，这一努力将使教师拥有工具，以识别并经常监测脱离科学学习的早期指标，并潜在地提高未被充分代表的学生群体的科学兴趣，并进一步使STEM工作队伍多样化。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The ability to learn concepts, especially for science and math (STEM) based disciplines, is impacted by educators who inspire, motivate, and create supportive environments and teaching methodologies which lower the entry barrier for students learning STEM subjects. Teaching resources nationwide have historically been constrained as STEM based science content for education expands with increasing student diversity based on prior science exposure in the classroom. A key aspect of student learning is to assess the quality of human communications between student-and-student as well as teacher-and-student. In STEM learning, students who are able to ask the right questions, know what they understand as well as what they need help with, allows educators to structure their teaching methods to help students overcome learning challenges. However, to date, it has been virtually impossible to collect and measure student-to-student or student-to-teacher voice communications in the classroom. Also, current speech technology is not sufficiently effective to overcome multi-speaker and naturalistic communications in classrooms. This project will develop classroom audio collection and measurement tools for students working together to solve problems, as well as teacher involvement with individual/groups of students. The audio collection solution includes both individual recorders on a sub-set of classroom students, as well as central smart speaker microphone collection units within each student group. Computer programs will be developed to analyze who is speaking and when, as well as spot keywords of interest for STEM topics and learning assessment. Privacy is maintained, since audio analysis is focused on high level measures such as individual student word counts, anonymous tagging of each speaker, and connecting conversational turns between students and teachers. A teacher driven keyword set will be used to help measure which students are having problems understanding concepts. These individual communication measured terms will be integrated into a dashboard display, to empower teachers with easy to use feedback on student engagement for STEM learning. The project has the potential to improve the ability to assess learning through classroom communications, and potentially help teachers better direct their time/expertise more efficiently to improve STEM learning for students. This project will develop ways to assess learning in classrooms by measuring the quality of human communication engagement between students-and-peers as well as teachers-and-students. Research has shown that learning is improved if there is dynamic interaction between student-to-student and student-to-teacher in voice communications. The project introduces personal recorders in the classroom to capture voice interactions during the entire day. Next, these multi-microphone recording streams are pooled together, where speech and language processing algorithms will be formulated to perform "audio diarization" - the process of determining "who spoke, what, and when", with potential keywords of interest based on classroom topics identified. The diarization output will drive the formulation of metrics to assess communication engagement. Communication based features derived from individual audio streams (word count, talk time, turn-taking, keyword profile) will be extracted on a per student basis through audio diarization. Next, this information flow will be used to develop class based group dynamics. This solution represents an approach for teachers to monitor student engagement over time in science activity areas, helping teachers identify students who are not verbally engaged in science discourse and quickly assess the impact of changes in classroom practices to improve learning. A number of technology challenges will be addressed for automatic audio stream based voice processing of naturalistic audio data using speech activity detection, speaker diarization based on machine learning models, and keyword spotting for science topic identification and tracking. These research aims will be assessed in classroom settings with teacher feedback on the effectiveness of the resulting solutions. The resulting speech technology advancements would offer new opportunities for future smart classrooms for voice assessment for teachers to better assess student involvement in science vs. infrequent traditional standardized testing. Ultimately, this effort will equip teachers with tools to identify and frequently monitor early indicators of disengagement in science learning, and potentially increase science interest by under-represented student populations and further diversify the STEM workforce.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.