Collaborative Research: Adaptive explicit and implicit feedback in second language pronunciation training

超过100万名国际学生在美国大学学习，其中大多数在STEM领域学习。所有人都需要用英语交流，这需要发音清晰。传统的看法是，简单地沉浸在说英语的环境中，随着时间的推移，会改善发音。然而，研究否定了这一观点：如果没有明确侧重于发音的教学(例如，Bad-Bed中的元音/ɛ/-/æ/)，学习者在新环境中只可能在第一年内取得进步，此后需要进行教学。不幸的是，面对面的发音教学很少，因此计算机辅助发音训练(CAPT)成为发音训练的最佳选择。CAPT课程很常见，但都有一个严重的弱点，就是不能向学习者提供有效的反馈。这项工作将检验CAPT中两种互补形式的发音反馈的有用性：显性反馈(侧重于向学习者提供关于发音错误的位置和性质的准确指导)和隐性反馈(依赖于学习者感知其错误发音的能力)。特别是，调查人员将开发错误发音检测算法，以突出学习者语音中的错误，他们将创建口音转换算法，可以为学习者生成个性化的语音样本：他们自己的声音可以产生母语语音。这两种形式的发音反馈最终将被集成到CAPT系统中，该系统自动适应学习者当前的发音表现，以最大限度地提高学习者的精确度。该研究在开发机器学习算法方面具有技术创新，可以同时解决口音转换和发音错误检测方面的挑战。在学习方面，本研究试图确定在语音学习的不同阶段，内隐反馈和外显反馈是否有效，从而最大限度地促进学习。最后，研究将语音技术和发音训练结合起来，发挥各自的优势。我们的目标是，建议的系统可以在没有教师参与的情况下被自主学习者成功使用，从而使个性化发音培训在规模上可行。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Over one million international students study at US universities, and the majority study in STEM fields. All need to communicate in English, which requires intelligible pronunciation. The conventional wisdom is that simple immersion in the English-speaking environment will, over time, improve pronunciation. Research, however, rejects this view: without instruction that is explicitly focused on pronunciation (e.g., the vowels /ɛ/-/æ/ as in bad-bed), learners are only likely to improve within the first year in the new environment, and instruction is needed after that. Unfortunately, face-to-face pronunciation instruction is infrequent, thus making computer-assisted pronunciation training (CAPT) the best option for pronunciation training. CAPT programs are common, but share a critical weakness of not providing effective feedback to the learner. This work will examine the usefulness of two complementary forms of pronunciation feedback in CAPT: explicit feedback (focused on delivering precise instruction to the learner about the location and nature of pronunciation errors), and implicit feedback (relying on the learner’s ability to perceive their mispronunciations). In particular, the investigators will develop mispronunciation-detection algorithms that can highlight errors in the learner’s speech, and they will create accent-conversion algorithms that can generate personalized speech samples for the learner: their own voice producing native-speech. These two forms of pronunciation feedback will ultimately be integrated into a CAPT system that automatically adapts to the learner’s current pronunciation performance to maximize the benefits for the learner as they develop their accuracy.This research is technologically innovative in developing machine-learning algorithms to simultaneously solve challenges in accent conversion and mispronunciation detection. In regard to learning, the research seeks to identify when implicit and explicit feedback are effective within different stages of pronunciation learning so as to maximize learning. Finally, the research integrates speech technology and pronunciation training to leverage their individual strengths. Our goal is that the proposed system can be successfully used by autonomous learners without involvement of instructors, thus making personalized pronunciation training feasible at scale.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.