EAGER: Collaborative Research: Second Language Speech Production: Formulation of Objective Speech Intelligibility Measures and Learner-Specific Feedback

这个早期概念探索性研究资助（EAGER）项目的重点是探索和开发一种新的基于语音、语言和感知的测量方法，以客观地评估第二语言（L2）语音产生的语音可理解性，并提供有效的学习者特定反馈。随着英语作为一门国际语言的兴起，基于可理解性的成功交流已经被强调过了母语口音。然而，第二语言教师经常担心学习者的发音进步缓慢或停滞不前。这个问题的几个主要原因可能包括在感知上辨别学习者言语变化的困难，以及在没有任何学习者特定的可理解性评估的情况下解释学习者的言语模式的困难。今天，教师没有系统的方法来评估每个学生的语音变化，学生也无法监控和跟踪与他们的发音学习进展相关的反馈。因此，本文提出了一种探索性和变革性的测量语音清晰度的方法，为教师和学习者提供客观和个性化的反馈。这一探索性项目是由EAGER资助的，目的是建立一个基线工作框架，用于创建可操作的目标措施，并为教师和学习者提供概念验证评估反馈。这种方法将帮助教师衡量学习者的理解水平，并允许学习者随着时间的推移逐步自我调节他们的学习进度。这项长期创新预计将使来自非英语国家的熟练美国专业人士受益，他们在各种STEM（科学、技术、工程和数学）领域工作。此外，这个跨学科项目为语言教育、应用语言学、计算机工程和语音技术领域的研究生和本科生提供了各种实践培训和经验的机会。本项目探讨了一种基于非母语者的多种个体言语测量来评估言语交流中的可理解性的想法。这些想法目前还处于非常早期的发展阶段，而且大部分的研究想法还没有经过测试。为了建立潜在的个体语音产生可理解度度量的基础真实性，必须用证据验证这种可理解度反馈方法的实施和可行性。通过采用先进的基于机器学习的基于自动语音识别的口音分类技术，研究小组计划通过一套可操作的和有区别的客观语音可理解性措施，为学习者提供可测量的语音属性信息。当前的创新建立在语言技能习得理论和功能分析语言学方法的基础上，认为外显和元语言反馈在推动学习者在二语发展中起着关键作用。视觉是通过正在进行的基于听觉的神经图和频谱图正交多项式测量来实现的，这些测量利用学习者的无约束语音表达来预测语音可理解性。该项目将有助于对二语可理解语音构成的科学认识，理解个性化客观语音可理解性反馈如何影响二语语音发展，并创建语音/听觉/信号处理措施的基础集合，以及ASR/DNN驱动的措施，以评估说话者的可理解性，并确定在二语学习环境中实施该技术的有效方法。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This Early-concept Grants for Exploratory Research (EAGER) project focuses on exploring and developing a novel operational collection of speech, language and perception-based measures to objectively assess speech intelligibility for second language (L2) speech production, as well as providing effective learner-specific feedback. With the rise of English as an international language, intelligibility-based successful communication has been emphasized over native-like accents. However, L2 teachers often raise concerns about learners’ slow or stagnant pronunciation progress. Several primary reasons for this problem may include difficulties in perceptually discerning changes in learners’ speech and interpreting learners’ speech patterns without any learner-specific intelligibility assessment profile. Today, teachers have no systematic way to assess each student’s speech changes, nor can students monitor and track feedback related to their pronunciation learning progression. Therefore, an exploratory and transformative method is introduced for measuring speech intelligibility that provides both teachers and learners with objective and individualized feedback. This exploratory project is proposed for EAGER funding in order to establish a baseline working framework for operational objective measure creation, and proof-of-concept assessment feedback for teachers and learners. This approach will help teachers gauge learners’ intelligibility levels and allow learners to self-regulate their learning progress incrementally over time. The long-term innovation is expected to benefit skilled US professionals from non-English speaking countries, who work in various STEM (science, technology, engineering, and mathematics) fields. Additionally, this interdisciplinary project provides various opportunities for hands-on training and experience for both graduate and undergraduate students in the fields of language education, applied linguistics, computer engineering, and speech technology.This project explores an idea to assess intelligibility in speech communications based on multiple individual speech measures for non-native speakers. The ideas are currently in their very early stages of development, and a large portion of the research ideas are untested. In order to establish the ground truth of potential individual speech production intelligibility measures, the implementation and feasibility of this intelligibility feedback approach must be validated with evidence. By employing advanced Automatic Speech Recognition-based accent classification technology based on machine learning, the team of researchers plan to provide learners with measured speech property information through operational and a discriminating set of objective speech intelligibility measures. The current innovation builds on language skill acquisition theory with a functional analytic-linguistic approach, arguing that explicit and metalinguistic feedback plays a pivotal role in moving learners forward in their L2 development. The vision is enabled by on-going research on auditory-based neurogram and spectrogram orthogonal polynomial measures that predict speech intelligibility, employing the learners’ unconstrained speech utterances. The project will contribute to the scientific knowledge of what constitutes L2 intelligible speech, understanding how individualized objective speech intelligibility feedback affects L2 speech development, and creating a foundational collection of speech/auditory/signal processing measures as well as ASR/DNN driven measures that assess a speaker’s intelligibility and identify efficient ways of implementing this technology in L2 learning contexts.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.