US Egypt Cooperative Research: Computer Aided Pronunciation Learning Application

这一合作研究项目由埃及罗格斯大学的艾哈迈德·埃尔伽马尔博士和埃及开罗大学的谢里夫·阿布杜博士共同实施，目的是创建一个自动化系统，输入语言学习者的发音以及他们说话时面部的动态图像。利用这一信息，拟议的系统将评估用户是否对目标语言中的特定单词或短语产生了准确的发音。然后，系统会就用户的发音质量向用户提供具体的反馈，并提出改进建议。具体地说，这项研究将需要分析非英语和阿拉伯语使用者出现的发音错误。研究人员预计，该语音识别系统将足够强大，能够承受用户使用非母语时可能出现的大量发音错误。此外，该系统将被设计为为用户分配通过或不及格的分数-S的发音，检测错误发生的位置，并对错误进行分类，以便就如何改变发音给出足够的反馈。为了建立这样一个系统，将开发新的算法，将视觉线索和音频线索结合起来，用于错误发音检测任务。这将通过部署嘴唇跟踪算法并通过个人用户的风格相关模型研究嘴唇运动和语音之间的相关性来实现。研究计划还将收集母语和非母语阿拉伯语和英语使用者的大型音素视听数据库。最初开发英语和阿拉伯语发音错误检测的意义在于，这两种语言都有丰富的语音，而且彼此不同，这导致了错误发音检测具有广泛的工作频谱的假设。高效的语音训练系统的社会效益是显著的。除了语言学习的许多好处外，这种合作研究的预期结果是一种工具，将成为任何计算机辅助语言学习(CALL)系统的核心组件之一。这样的结果可以为听力受损的人提供一种具有成本效益和个性化的工具，在那里获得可接受的发音可能是一种挑战。

This collaborative research project is being undertaken by Dr. Ahmed Elgammal, Rutgers University, and Dr. Sherif Abdou, Cairo University in Egypt, in order to create an automated system that will input pronunciations from language learners along with dynamic images of their faces as they speak. Using this information, the proposed system will assess whether the user has produced an accurate pronunciation of a particular word or phrase in the target language. The system will then offer specific feedback to the user regarding the quality of their pronunciation, with suggestions for improvement. Specifically, the research will entail analyzing pronunciation errors that occur for non-native speakers of English and Arabic.The researchers anticipate that this speech recognition system will be robust enough to withstand large mispronunciations that may occur when the user has a non-native tongue. Additionally, the system will be designed to assign a pass or fail score to the user?s utterance, detect where the error occurred, and classify the error in order to give adequate feedback on how the pronunciation should be altered. In order to create such a system, novel algorithms will be developed for combining visual cues with audio cues for the task of mispronunciation detection. This will be achieved by deploying lip tracking algorithms and studying the correlation between lip movement and speech through style-dependent models of individual users. The research plan will also result in collection of a large audio-visual database of the phonemes by native and non-native Arabic and English speakers. The significance of initially developing the mispronunciation detection for English and Arabic is that these two languages are both phonetically rich and different from one another, leading to an assumption that the mispronunciation detection has a wide working spectrum.The societal benefits of efficient speech training systems are significant. In addition to numerous benefits for language learning, the expected outcome of this collaborative research is a tool that would be one of the core components for any Computer Aided Language Learning (CALL) system. Such an outcome could provide a cost effective and personalized tool for hearing-impaired speakers where the acquisition of an acceptable pronunciation can be challenging.