Collaborative Research: Adaptive explicit and implicit feedback in second language pronunciation training

超过100万名国际学生在美国大学学习，其中大多数在STEM(科学、技术、工程和数学)领域学习。所有人都需要用英语交流，这需要发音清晰。传统的看法是，简单地沉浸在说英语的环境中，随着时间的推移，会改善发音。然而，研究否定了这一观点：如果没有明确侧重于发音的教学(例如，Bad中的元音与Bed中的元音)，学习者在新环境中只有可能在第一年内取得进步，此后需要进行教学。不幸的是，面对面的发音教学很少，因此计算机辅助发音训练(CAPT)成为大规模发音训练的最佳选择。CAPT课程很常见，但都有一个严重的弱点，就是不能向学习者提供有效的反馈。本项目将考察CAPT中两种互补形式的发音反馈的有用性：显性反馈(侧重于向学习者提供关于发音错误的位置和性质的准确指导)和隐性反馈(依赖于学习者感知其错误发音的能力)。为此，研究人员将开发错误发音检测算法，以突出学习者语音中的错误(显式反馈)，他们将创建口音转换算法，可以为学习者生成个性化的语音样本：他们自己的声音产生母语语音(隐式反馈)。具体地说，调查人员将开发一个机器学习框架，同时解决发音错误检测和口音转换的问题。机器学习的研究表明，只要任务相关，同时求解多个任务可以提高学习效率和泛化性能。因此，这一奖项提供了在这两个问题上推进最先进技术的潜力。两种形式的发音反馈(显式和隐式)将被整合到CAPT系统中，该系统自动适应学习者的当前表现，以在学习者发展其准确性的同时最大限度地为学习者带来好处。该系统将使用自适应辅导算法来指导学习者首先建立对他们发音困难的感知，然后提高他们的发音准确性。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Over one million international students study at US universities, and the majority study in STEM (Science, Technology, Engineering, and Mathematics) fields. All need to communicate in English, which requires intelligible pronunciation. The conventional wisdom is that simple immersion in the English-speaking environment will, over time, improve pronunciation. Research, however, rejects this view: without instruction that is explicitly focused on pronunciation (e.g., the vowels in "bad" vs. "bed"), learners are only likely to improve within the first year in the new environment, and instruction is needed after that. Unfortunately, face-to-face pronunciation instruction is infrequent, thus making computer-assisted pronunciation training (CAPT) the best option for pronunciation training at scale. CAPT programs are common, but share a critical weakness of not providing effective feedback to the learner. This project will examine the usefulness of two complementary forms of pronunciation feedback in CAPT: explicit feedback (focused on delivering precise instruction to the learner about the location and nature of pronunciation errors) and implicit feedback (relying on the learner's ability to perceive their mispronunciations). To this end, the investigators will develop mispronunciation-detection algorithms that can highlight errors in the learner's speech (explicit feedback), and they will create accent-conversion algorithms that can generate personalized speech samples for the learner: their own voice producing native-speech (implicit feedback). Specifically, the investigators will develop a machine-learning framework to solve the problems of mispronunciation detection and accent conversion simultaneously. Research in machine learning shown that solving multiple tasks simultaneously can improve learning efficiency and generalization performance, so long as the tasks are related. Thus, this award offers the potential to advance the state-of-the-art in both problems. The two forms of pronunciation feedback (explicit and implicit) will be integrated into a CAPT system that automatically adapts to the learner's current performance to maximize the benefits for the learner as they develop their accuracy. The system will use an adaptive tutoring algorithm to guide the learner in first establishing perception of their pronunciation difficulties and then enhancing their production accuracy.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.