Collaborative Research: Adaptive explicit and implicit feedback in second language pronunciation training

点击翻译按钮获取中文摘要

Over one million international students study at US universities, and the majority study in STEM fields. All need to communicate in English, which requires intelligible pronunciation. The conventional wisdom is that simple immersion in the English-speaking environment will, over time, improve pronunciation. Research, however, rejects this view: without instruction that is explicitly focused on pronunciation (e.g., the vowels /ɛ/-/æ/ as in bad-bed), learners are only likely to improve within the first year in the new environment, and instruction is needed after that. Unfortunately, face-to-face pronunciation instruction is infrequent, thus making computer-assisted pronunciation training (CAPT) the best option for pronunciation training. CAPT programs are common, but share a critical weakness of not providing effective feedback to the learner. This work will examine the usefulness of two complementary forms of pronunciation feedback in CAPT: explicit feedback (focused on delivering precise instruction to the learner about the location and nature of pronunciation errors), and implicit feedback (relying on the learner’s ability to perceive their mispronunciations). In particular, the investigators will develop mispronunciation-detection algorithms that can highlight errors in the learner’s speech, and they will create accent-conversion algorithms that can generate personalized speech samples for the learner: their own voice producing native-speech. These two forms of pronunciation feedback will ultimately be integrated into a CAPT system that automatically adapts to the learner’s current pronunciation performance to maximize the benefits for the learner as they develop their accuracy.This research is technologically innovative in developing machine-learning algorithms to simultaneously solve challenges in accent conversion and mispronunciation detection. In regard to learning, the research seeks to identify when implicit and explicit feedback are effective within different stages of pronunciation learning so as to maximize learning. Finally, the research integrates speech technology and pronunciation training to leverage their individual strengths. Our goal is that the proposed system can be successfully used by autonomous learners without involvement of instructors, thus making personalized pronunciation training feasible at scale.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.