SBIR Phase I: A language learning app based on sound and mouth movements

这个小企业创新研究 (SBIR) 第一阶段项目的更广泛/商业影响是通过将面部和唇形识别与声音分析相结合来推进新的语言学习。创造声音的视觉方面对于掌握发音至关重要，这是学习外语甚至提高母语的重大障碍之一。当前的语言学习方法往往无法帮助学习者达到口语熟练程度，也无法提供现实生活中的语言使用体验。该语言学习平台旨在通过满足工作场所和学术环境中对多语言能力日益增长的需求，提供有效且引人入胜的语言学习体验来改变这一现状。当前的语言学习方法和应用程序往往无法培养口语和写作能力，而是专注于记忆和标准化测试。这位语言培训师通过提供对语音产生科学的见解来弥补这一差距。通过将口腔形状的视觉提示与听觉输入相结合，学习者可以掌握发音，这是语言习得的一项重大挑战。这项研究将包括获取近乎完美的语音文件用于机器学习模型训练、语音和视频文件的信号处理、机器学习模型的开发和比较、数据可视化开发、纳入移动测试套件以及初步测试。机器学习算法将利用从学生语音数据中提取的见解，为学习者提供高度针对性、微调的活动。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

The broader/commercial impact of this Small Business Innovation Research (SBIR) Phase I project is advancing new language learning by incorporating facial and lip recognition along with sound analysis. This visual aspect of creating sounds is vital for mastering pronunciation, one of the significant hurdles of learning a foreign language and even improving a native language. Current language learning methods often fall short in helping learners achieve speaking proficiency and fail to provide real-life language usage experiences. This language learning platform aims to change this by addressing the growing need for multi-language proficiency in workplaces and academic settings, providing an effective and engaging language learning experience.Current language learning methods and apps often fail to develop speaking and writing proficiency, focusing instead on memorization and standardized tests. This language trainer addresses this gap by offering insights into the science of speech production. By combining visual cues of oral shapes with auditory input, learners can master pronunciation, a significant challenge in language acquisition. This research will include obtaining near-perfect voice files for machine learning model training, signal processing of the voice and video files, development and comparison of machine learning models, data visualization development, incorporation into the mobile test suite, and preliminary testing. The machine learning algorithms will use the insights extracted from students' voice data to provide learners with highly targeted, fine-tuned activities.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.