Collaborative Research: Adaptive explicit and implicit feedback in second language pronunciation training

合作研究：第二语言发音训练中的自适应显式和隐式反馈

• 批准号: 2016959
• 负责人: Ricardo Gutierrez-Osuna
• 金额: $ 33.26万
• 依托单位: Texas A&M Engineering Experiment Station
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-09-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2016959&HistoricalAwards=false
• 关键词: Collaborative; Research; Adaptive; explicit; implicit

Over one million international students study at US universities, and the majority study in STEM fields. All need to communicate in English, which requires intelligible pronunciation. The conventional wisdom is that simple immersion in the English-speaking environment will, over time, improve pronunciation. Research, however, rejects this view: without instruction that is explicitly focused on pronunciation (e.g., the vowels /ɛ/-/æ/ as in bad-bed), learners are only likely to improve within the first year in the new environment, and instruction is needed after that. Unfortunately, face-to-face pronunciation instruction is infrequent, thus making computer-assisted pronunciation training (CAPT) the best option for pronunciation training. CAPT programs are common, but share a critical weakness of not providing effective feedback to the learner. This work will examine the usefulness of two complementary forms of pronunciation feedback in CAPT: explicit feedback (focused on delivering precise instruction to the learner about the location and nature of pronunciation errors), and implicit feedback (relying on the learner’s ability to perceive their mispronunciations). In particular, the investigators will develop mispronunciation-detection algorithms that can highlight errors in the learner’s speech, and they will create accent-conversion algorithms that can generate personalized speech samples for the learner: their own voice producing native-speech. These two forms of pronunciation feedback will ultimately be integrated into a CAPT system that automatically adapts to the learner’s current pronunciation performance to maximize the benefits for the learner as they develop their accuracy.This research is technologically innovative in developing machine-learning algorithms to simultaneously solve challenges in accent conversion and mispronunciation detection. In regard to learning, the research seeks to identify when implicit and explicit feedback are effective within different stages of pronunciation learning so as to maximize learning. Finally, the research integrates speech technology and pronunciation training to leverage their individual strengths. Our goal is that the proposed system can be successfully used by autonomous learners without involvement of instructors, thus making personalized pronunciation training feasible at scale.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

超过100万国际学生在美国大学学习，其中大多数在STEM领域学习。所有人都需要用英语交流，这需要清晰的发音。传统的观点认为，简单地沉浸在讲英语的环境中，随着时间的推移，会改善发音。然而，研究拒绝了这一观点：如果没有明确关注发音的教学（例如，元音//-//，如bad-bed中的元音），学习者在新环境中只可能在第一年内有所改善，之后需要指导。不幸的是，面对面的发音教学是罕见的，因此使计算机辅助发音训练（CAPT）的发音训练的最佳选择。CAPT程序很常见，但有一个关键的弱点，即不能向学习者提供有效的反馈。这项工作将研究两种互补形式的发音反馈CAPT的有用性：明确的反馈（侧重于提供精确的指令，学习者的位置和性质的发音错误），和隐式反馈（依赖于学习者的能力，感知他们的发音错误）。特别是，研究人员将开发能够突出学习者语音中错误的发音检测算法，他们将创建能够为学习者生成个性化语音样本的口音转换算法：他们自己的声音产生母语。这两种形式的发音反馈最终将被集成到一个CAPT系统，自动适应学习者当前的发音表现，以最大限度地提高学习者的利益，因为他们发展自己的accuracy.This research is technologically innovative in developing machine learning algorithms to simultaneously solve challenges in accent conversion and mispronunciation detection.在学习方面，本研究试图确定在语音学习的不同阶段内隐反馈和外显反馈何时有效，以最大限度地提高学习效果。最后，本研究整合语音技术与发音训练，以发挥其个人优势。我们的目标是，建议的系统可以成功地使用自主学习者没有教师的参与，从而使个性化的发音培训可行的规模。这个奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

项目成果

Decoupling Segmental and Prosodic Cues of Non-native Speech through Vector Quantization

• DOI: 10.21437/interspeech.2023-2202
• 发表时间: 2023-08
• 作者: Waris Quamer;Anurag Das;R. Gutierrez-Osuna

Zero-Shot Foreign Accent Conversion without a Native Reference

• DOI: 10.21437/interspeech.2022-10664
• 发表时间: 2022-09
• 作者: Waris Quamer;Anurag Das;John M. Levis;E. Chukharev-Hudilainen;R. Gutierrez-Osuna