Collaborative Research: CSL-MultiAD: Assessing Collaborative STEM Learning through Rich Information Flow based on Multi-Sensor Audio Diarization

协作研究：CSL-MultiAD：通过基于多传感器音频二值化的丰富信息流评估协作 STEM 学习

• 批准号: 1918032
• 负责人: John Hansen
• 金额: $ 45万
• 依托单位: University of Texas at Dallas
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-08-01 至 2024-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1918032&HistoricalAwards=false
• 关键词: Collaborative; Research; CSL; MultiAD; Assessing

The ability to learn concepts, especially for science and math (STEM) based disciplines, is impacted by educators who inspire, motivate, and create supportive environments and teaching methodologies which lower the entry barrier for students learning STEM subjects. Teaching resources nationwide have historically been constrained as STEM based science content for education expands with increasing student diversity based on prior science exposure in the classroom. A key aspect of student learning is to assess the quality of human communications between student-and-student as well as teacher-and-student. In STEM learning, students who are able to ask the right questions, know what they understand as well as what they need help with, allows educators to structure their teaching methods to help students overcome learning challenges. However, to date, it has been virtually impossible to collect and measure student-to-student or student-to-teacher voice communications in the classroom. Also, current speech technology is not sufficiently effective to overcome multi-speaker and naturalistic communications in classrooms. This project will develop classroom audio collection and measurement tools for students working together to solve problems, as well as teacher involvement with individual/groups of students. The audio collection solution includes both individual recorders on a sub-set of classroom students, as well as central smart speaker microphone collection units within each student group. Computer programs will be developed to analyze who is speaking and when, as well as spot keywords of interest for STEM topics and learning assessment. Privacy is maintained, since audio analysis is focused on high level measures such as individual student word counts, anonymous tagging of each speaker, and connecting conversational turns between students and teachers. A teacher driven keyword set will be used to help measure which students are having problems understanding concepts. These individual communication measured terms will be integrated into a dashboard display, to empower teachers with easy to use feedback on student engagement for STEM learning. The project has the potential to improve the ability to assess learning through classroom communications, and potentially help teachers better direct their time/expertise more efficiently to improve STEM learning for students. This project will develop ways to assess learning in classrooms by measuring the quality of human communication engagement between students-and-peers as well as teachers-and-students. Research has shown that learning is improved if there is dynamic interaction between student-to-student and student-to-teacher in voice communications. The project introduces personal recorders in the classroom to capture voice interactions during the entire day. Next, these multi-microphone recording streams are pooled together, where speech and language processing algorithms will be formulated to perform "audio diarization" - the process of determining "who spoke, what, and when", with potential keywords of interest based on classroom topics identified. The diarization output will drive the formulation of metrics to assess communication engagement. Communication based features derived from individual audio streams (word count, talk time, turn-taking, keyword profile) will be extracted on a per student basis through audio diarization. Next, this information flow will be used to develop class based group dynamics. This solution represents an approach for teachers to monitor student engagement over time in science activity areas, helping teachers identify students who are not verbally engaged in science discourse and quickly assess the impact of changes in classroom practices to improve learning. A number of technology challenges will be addressed for automatic audio stream based voice processing of naturalistic audio data using speech activity detection, speaker diarization based on machine learning models, and keyword spotting for science topic identification and tracking. These research aims will be assessed in classroom settings with teacher feedback on the effectiveness of the resulting solutions. The resulting speech technology advancements would offer new opportunities for future smart classrooms for voice assessment for teachers to better assess student involvement in science vs. infrequent traditional standardized testing. Ultimately, this effort will equip teachers with tools to identify and frequently monitor early indicators of disengagement in science learning, and potentially increase science interest by under-represented student populations and further diversify the STEM workforce.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

学习概念的能力，特别是科学和数学（STEM）为基础的学科，受到教育工作者的影响，他们激励，激励和创造支持性的环境和教学方法，降低学生学习STEM科目的入学门槛。全国范围内的教学资源在历史上一直受到限制，因为基于STEM的科学教育内容随着学生在课堂上先前科学接触的增加而增加。学生学习的一个关键方面是评估学生与学生以及教师与学生之间的人际沟通质量。在STEM学习中，能够提出正确问题的学生，知道他们理解什么以及他们需要什么帮助，允许教育工作者构建他们的教学方法，以帮助学生克服学习挑战。然而，迄今为止，在教室中收集和测量学生与学生或学生与教师的语音通信几乎是不可能的。此外，目前的语音技术还不足以有效地克服教室中的多扬声器和自然主义通信。 该项目将开发课堂音频收集和测量工具，供学生一起解决问题，以及教师与学生个人/团体的参与。音频收集解决方案包括教室学生子集上的单独录音机，以及每个学生组内的中央智能扬声器麦克风收集单元。将开发计算机程序来分析谁在说话，什么时候说话，以及发现STEM主题和学习评估的关键词。隐私得到维护，因为音频分析专注于高水平的措施，如个别学生的字数，每个发言者的匿名标记，并连接学生和教师之间的会话回合。教师驱动的关键字集将用于帮助衡量哪些学生在理解概念方面存在问题。这些单独的沟通测量术语将被集成到仪表板显示中，以使教师能够轻松使用关于学生参与STEM学习的反馈。该项目有可能提高通过课堂交流评估学习的能力，并有可能帮助教师更有效地指导他们的时间/专业知识，以改善学生的STEM学习。该项目将通过测量学生与同伴以及教师与学生之间的人际交往参与的质量来开发评估课堂学习的方法。研究表明，如果在语音通信中学生与学生和学生与教师之间存在动态交互，则学习会得到改善。该项目在教室中引入了个人录音机，以捕捉全天的语音交互。接下来，这些多麦克风记录流被汇集在一起，其中语音和语言处理算法将被制定为执行“音频日记”-确定“谁说话，什么，什么时候”的过程，以及基于课堂主题确定的潜在关键词。日志化产出将推动制定衡量标准，以评估沟通参与情况。从个人音频流（字数，通话时间，话轮转换，关键字配置文件）派生的通信为基础的功能将通过音频日记每个学生的基础上提取。接下来，这个信息流将被用来开发基于类的组动态。这个解决方案代表了一种方法，让教师监测学生参与随着时间的推移在科学活动领域，帮助教师确定谁是不口头参与科学话语的学生，并迅速评估课堂实践的变化，以改善学习的影响。许多技术挑战将被解决的自动音频流为基础的语音处理的自然音频数据使用语音活动检测，说话人日记的基础上机器学习模型，和关键字定位科学主题识别和跟踪。这些研究目标将在课堂环境中进行评估，教师对所产生的解决方案的有效性进行反馈。由此产生的语音技术进步将为未来的智能教室提供新的机会，以便教师更好地评估学生对科学的参与，而不是不频繁的传统标准化测试。 最终，这一努力将为教师提供工具，以识别和经常监测科学学习脱离的早期指标，并有可能提高代表性不足的学生群体的科学兴趣，并进一步使STEM劳动力多样化。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

项目成果

Child vs Adult Speaker Diarization of naturalistic audio recordings in preschool environment using Deep Neural Networks

使用深度神经网络对学前环境中的自然录音进行儿童与成人说话者的分类

• 发表时间: 2021
• 期刊: ASEE 2021 Gulf-Southwest Annual Conference
• 作者: Kothalkar, P.;Hansen, J.H.L.;Buzhardt, J.;Irvin, D.;Rous, B.
• 通讯作者: Rous, B.

Block-Based High Performance CNN Architectures for Frame-Level Overlapping Speech Detection

• DOI: 10.1109/taslp.2020.3036237
• 发表时间: 2021-01-01
• 期刊: IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING
• 影响因子: 5.4
• 作者: Yousefi, Midia;Hansen, John H. L.
• 通讯作者: Hansen, John H. L.

Domain Expansion for End-to-End Speech Recognition: Applications for Accent/Dialect Speech

• DOI: 10.1109/taslp.2022.3233238
• 发表时间: 2023
• 期刊: IEEE/ACM Transactions on Audio, Speech, and Language Processing
• 作者: Shahram Ghorbani;J. Hansen

Can Smartphones be a cost-effective alternative to LENA for Early Childhood Language Intervention?

• DOI: 10.21437/s4sg.2022-3
• 发表时间: 2022-09
• 期刊: 1st Workshop on Speech for Social Good (S4SG)
• 作者: Satwik Dutta;Jacob C. Reyna;J. Buzhardt;Dwight W. Irvin;John H. L. Hansen

Quantifying Engagement in Preschool Classrooms - Conversational Turn-Taking & Topic Initiations

量化学前班课堂的参与度 - 对话轮流

• DOI: 10.18260/1-2--39202
• 发表时间: 2022
• 期刊: 2022 ASEE Gulf Southwest Annual Conference Proceedings
• 作者: Sarah A. Tao;Satwik Dutta;Yagmur Seven;Dwight Irvin;J. Buzhardt;John H. L. Hansen
• 通讯作者: John H. L. Hansen