EAGER: Collaborative Research: Second Language Speech Production: Formulation of Objective Speech Intelligibility Measures and Learner-Specific Feedback

EAGER：协作研究：第二语言语音生成：客观语音清晰度测量和学习者特定反馈的制定

• 批准号: 2140415
• 负责人: John Hansen
• 金额: $ 13.5万
• 依托单位: University of Texas at Dallas
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2140415&HistoricalAwards=false
• 关键词: EAGER; Collaborative; Research; Second; Language

This Early-concept Grants for Exploratory Research (EAGER) project focuses on exploring and developing a novel operational collection of speech, language and perception-based measures to objectively assess speech intelligibility for second language (L2) speech production, as well as providing effective learner-specific feedback. With the rise of English as an international language, intelligibility-based successful communication has been emphasized over native-like accents. However, L2 teachers often raise concerns about learners’ slow or stagnant pronunciation progress. Several primary reasons for this problem may include difficulties in perceptually discerning changes in learners’ speech and interpreting learners’ speech patterns without any learner-specific intelligibility assessment profile. Today, teachers have no systematic way to assess each student’s speech changes, nor can students monitor and track feedback related to their pronunciation learning progression. Therefore, an exploratory and transformative method is introduced for measuring speech intelligibility that provides both teachers and learners with objective and individualized feedback. This exploratory project is proposed for EAGER funding in order to establish a baseline working framework for operational objective measure creation, and proof-of-concept assessment feedback for teachers and learners. This approach will help teachers gauge learners’ intelligibility levels and allow learners to self-regulate their learning progress incrementally over time. The long-term innovation is expected to benefit skilled US professionals from non-English speaking countries, who work in various STEM (science, technology, engineering, and mathematics) fields. Additionally, this interdisciplinary project provides various opportunities for hands-on training and experience for both graduate and undergraduate students in the fields of language education, applied linguistics, computer engineering, and speech technology.This project explores an idea to assess intelligibility in speech communications based on multiple individual speech measures for non-native speakers. The ideas are currently in their very early stages of development, and a large portion of the research ideas are untested. In order to establish the ground truth of potential individual speech production intelligibility measures, the implementation and feasibility of this intelligibility feedback approach must be validated with evidence. By employing advanced Automatic Speech Recognition-based accent classification technology based on machine learning, the team of researchers plan to provide learners with measured speech property information through operational and a discriminating set of objective speech intelligibility measures. The current innovation builds on language skill acquisition theory with a functional analytic-linguistic approach, arguing that explicit and metalinguistic feedback plays a pivotal role in moving learners forward in their L2 development. The vision is enabled by on-going research on auditory-based neurogram and spectrogram orthogonal polynomial measures that predict speech intelligibility, employing the learners’ unconstrained speech utterances. The project will contribute to the scientific knowledge of what constitutes L2 intelligible speech, understanding how individualized objective speech intelligibility feedback affects L2 speech development, and creating a foundational collection of speech/auditory/signal processing measures as well as ASR/DNN driven measures that assess a speaker’s intelligibility and identify efficient ways of implementing this technology in L2 learning contexts.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

这个早期概念的探索性研究（EAGER）项目的赠款的重点是探索和开发一个新的语音，语言和感知为基础的措施，以客观地评估第二语言（L2）语音生产的语音清晰度操作集合，以及提供有效的学习者特定的反馈。随着英语作为一种国际语言的崛起，基于可理解性的成功沟通已经超过了类似母语的口音。然而，第二语言教师经常对学习者缓慢或停滞的发音进度表示担忧。这个问题的几个主要原因可能包括在没有任何学习者特定的可懂度评估简档的情况下难以感知地辨别学习者语音的变化和解释学习者的语音模式。今天，教师没有系统的方法来评估每个学生的语音变化，学生也不能监控和跟踪与他们的发音学习进展相关的反馈。因此，一个探索性和变革性的方法来衡量语音可懂度，为教师和学习者提供客观和个性化的反馈。这一探索性项目是为EAGER基金提出的，目的是建立一个基线工作框架，为教师和学习者创建操作目标措施和概念验证评估反馈。这种方法将有助于教师衡量学习者的可理解程度，并允许学习者随着时间的推移逐步自我调节他们的学习进度。这项长期创新预计将使来自非英语国家的熟练美国专业人士受益，他们在各种STEM（科学，技术，工程和数学）领域工作。此外，该跨学科项目为语言教育、应用语言学、计算机工程和语音技术领域的研究生和本科生提供了各种实践培训和经验的机会。该项目探讨了一种基于多个个体语音测量的非母语人士语音通信可懂度评估的想法。这些想法目前处于发展的早期阶段，大部分研究想法未经测试。为了建立潜在的个人语音生产可懂度措施的地面真相，这种可懂度反馈方法的实施和可行性必须用证据进行验证。通过采用基于机器学习的先进的基于自动语音识别的口音分类技术，研究人员计划通过一组可操作的和有区别的客观语音清晰度测量，为学习者提供测量的语音属性信息。目前的创新建立在语言技能习得理论与功能分析语言学的方法，认为明确的和元语言反馈在推动学习者在他们的第二语言发展的关键作用。该愿景是通过正在进行的基于神经网络的神经图和声谱图正交多项式测量的研究来实现的，这些测量预测语音可懂度，采用学习者的无约束语音话语。该项目将有助于科学知识的什么构成L2可懂语音，了解如何个性化的客观语音可懂度反馈影响L2语音发展，并创建语音/听觉/信号处理措施以及ASR/DNN驱动的措施，评估扬声器的可理解性，并确定在第二语言学习环境中实施这项技术的有效方法。基金会的使命是履行其法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Assessment of Non-Native Speech Intelligibility using Wav2vec2-based Mispronunciation Detection and Multi-level Goodness of Pronunciation Transformer

• DOI: 10.21437/interspeech.2023-2371
• 发表时间: 2023-08
• 作者: R. Shekar;Mu Yang;K. Hirschi;Stephen Looney;Okim Kang;J. Hansen

Mobile-Assisted Pronunciation Training With Adult Esol Learners: Background, Acceptance, Effort, and Accuracy

成人 ESOL 学习者的移动辅助发音训练：背景、接受程度、努力程度和准确性

• DOI: 10.31274/psllt.13272
• 发表时间: 2022
• 期刊: PSLLT: Proceedings of 12th Pronunciation in Second Language Learning and Teaching Conference
• 作者: Hirschi, Kevin;Kang, Okim;Hansen, John;Cucchiarini, Catia;Strik, Helmer
• 通讯作者: Strik, Helmer

Mobile-assisted pronunciation training with limited English proficiency: Learner background and technology attitude

英语水平有限的移动辅助发音训练：学习者背景和技术态度

• 发表时间: 2021
• 期刊: PSLLT-2021: Pronunciation in Second Language Learning and Teaching Conf.
• 作者: Hirschi, K.;Kang, O.;Cucchiarini, C.;Hansen, J.H.L.;Strik, H.
• 通讯作者: Strik, H.

What Can an Accent Identifier Learn? Probing Phonetic and Prosodic Information in a Wav2vec2-based Accent Identification Model

口音识别器能学到什么？

• DOI: 10.21437/interspeech.2023-2254
• 发表时间: 2023
• 期刊: ISCA INTERSPEECH-2023
• 作者: Yang, Mu;Shekar, Ram C.;Kang, Okim;Hansen, John H.
• 通讯作者: Hansen, John H.