Collaborative Research: Adaptive explicit and implicit feedback in second language pronunciation training

超过100万国际学生在美国大学学习，大多数在STEM（科学，技术，工程和数学）领域学习。所有人都需要用英语交流，这需要清晰的发音。传统的观点认为，简单地沉浸在讲英语的环境中，随着时间的推移，会改善发音。然而，研究拒绝了这一观点：如果没有明确关注发音的教学（例如，“bad”vs.“bed”中的元音），学习者在新环境中只可能在第一年内有所提高，之后需要指导。不幸的是，面对面的发音教学是罕见的，因此使计算机辅助发音训练（CAPT）的最佳选择，发音训练的规模。CAPT程序很常见，但有一个关键的弱点，即不能向学习者提供有效的反馈。本项目将研究CAPT中两种互补形式的发音反馈的有用性：明确的反馈（侧重于向学习者提供关于发音错误的位置和性质的精确指导）和隐含的反馈（依赖于学习者感知其错误发音的能力）。为此，研究人员将开发能够突出学习者语音中错误的发音检测算法（显式反馈），并创建能够为学习者生成个性化语音样本的口音转换算法：他们自己的声音产生母语语音（隐式反馈）。具体来说，研究人员将开发一个机器学习框架，以同时解决发音错误检测和口音转换的问题。 机器学习的研究表明，同时解决多个任务可以提高学习效率和泛化性能，只要任务是相关的。因此，该奖项提供了在这两个问题上推进最先进技术的潜力。这两种形式的发音反馈（显性和隐性）将被整合到CAPT系统中，该系统将自动适应学习者当前的表现，以最大限度地提高学习者的准确性。该系统将使用自适应辅导算法，引导学习者首先建立对发音困难的感知，然后提高他们的发音准确性。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Over one million international students study at US universities, and the majority study in STEM (Science, Technology, Engineering, and Mathematics) fields. All need to communicate in English, which requires intelligible pronunciation. The conventional wisdom is that simple immersion in the English-speaking environment will, over time, improve pronunciation. Research, however, rejects this view: without instruction that is explicitly focused on pronunciation (e.g., the vowels in "bad" vs. "bed"), learners are only likely to improve within the first year in the new environment, and instruction is needed after that. Unfortunately, face-to-face pronunciation instruction is infrequent, thus making computer-assisted pronunciation training (CAPT) the best option for pronunciation training at scale. CAPT programs are common, but share a critical weakness of not providing effective feedback to the learner. This project will examine the usefulness of two complementary forms of pronunciation feedback in CAPT: explicit feedback (focused on delivering precise instruction to the learner about the location and nature of pronunciation errors) and implicit feedback (relying on the learner's ability to perceive their mispronunciations). To this end, the investigators will develop mispronunciation-detection algorithms that can highlight errors in the learner's speech (explicit feedback), and they will create accent-conversion algorithms that can generate personalized speech samples for the learner: their own voice producing native-speech (implicit feedback). Specifically, the investigators will develop a machine-learning framework to solve the problems of mispronunciation detection and accent conversion simultaneously. Research in machine learning shown that solving multiple tasks simultaneously can improve learning efficiency and generalization performance, so long as the tasks are related. Thus, this award offers the potential to advance the state-of-the-art in both problems. The two forms of pronunciation feedback (explicit and implicit) will be integrated into a CAPT system that automatically adapts to the learner's current performance to maximize the benefits for the learner as they develop their accuracy. The system will use an adaptive tutoring algorithm to guide the learner in first establishing perception of their pronunciation difficulties and then enhancing their production accuracy.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.