Collaborative Research: Improving Techniques of Automatic Speech Recognition and Transfer Learning using Documentary Linguistic Corpora

越来越多的计算工具，特别是自动语音识别（即语音到文本的转换），被用于促进和调解各行各业的沟通。医生对着他们的电脑说话，电脑会将他们的讲话转录成清晰的书面摘要;在线虚拟助手在各种情况下的支持网络中无处不在;最终用户越来越希望他们的讲话能被手机、导航设备和Alexa等工具理解、处理和执行。然而，这种机制的创建目前依赖于大量的训练数据（语音和文本），而这些数据仅适用于主要语言。当只有10小时的转录音频可用时，开发语音识别系统是相当具有挑战性的。解决这个问题的一种方法是通过迁移学习，其中语音识别器在一种濒危语言（50小时的转录音频）的相对大量的数据上进行训练，然后扩展到仅开发一小部分材料的相关语言（10小时的转录音频和90小时的未转录音频）。该项目的目标既是理论性的，也是实质性的。首先，该项目将推动低资源语言的自然语言处理的发展，并建立一个将其扩展到其他相关语言的协议。实质上，该项目将为五种相关语言制作一个前所未有的转录音频语料库，促进理论和描述语言学家对这些语言的比较研究。最先进的自动语音识别（ASR）取决于材料语料库（具有时间编码的录音）的存在和人工智能系统的应用，该系统利用神经网络通过解释原始数据来复制人类学习。本项目采用所谓的“端到端神经网络”。“有效地，人工神经网络呈现输入数据（声学语音信号）和准备的最终结果（转录），并学习实现相同的结果。为了实现这一点，原始语料库被分为训练集（约80%）、验证集（约10%）和测试集（约10%）。对于濒危语言文档，目标不仅仅是ASR系统的准确性，而且还减少了人工努力，以实现高度准确的时间编码翻译，并将其作为目标语言的永久记录存档。该项目团队已经为一种语音学上困难的音调语言开发了一个高度准确的系统（字符错误率为8%），并将产生准确的时间编码转录所需的人力减少了75%（从人类从头开始需要40小时到人类校对ASR生成的转录所需的9小时）。在这个项目中，同一个团队将为一种形态复杂的粘着语言探索ASR策略，希望达到同样的准确度，并减少人类的努力。该项目还将解决最先进的ASR面临的另一个挑战：将为一种语言开发的有效系统转移到低资源，几乎没有文档的相关语言。如果该项目取得成功，它将成为与其他语文和语文组进行类似努力的典范。 这些数据和研究结果将在宾夕法尼亚大学的语言数据联盟和俄克拉荷马州大学的萨姆·诺布尔俄克拉荷马州自然历史博物馆提供。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

Increasingly, computational tools, particularly automatic speech recognition (that is, the conversion of speech to text), are used to facilitate and mediate communication in all walks of life. Doctors speak into their computers, which transcribe their speech into legible written summaries; online virtual assistants have become ubiquitous in support networks in a wide range of situations; and end users increasingly expect their speech to be understood, processed, and acted upon by cell phones, navigation devices, and tools such as Alexa. The creation of such mechanisms, however, is currently dependent upon a large amount of training data (speech and text) that is only available for major languages. It is quite challenging to develop speech recognition systems when only 10 hours of transcribed audio is available. One way of addressing this problem is through transfer learning, in which a speech recognizer is trained on a relatively large amount of data for one endangered language ( 50 hours of transcribed audio) is then extended to related languages for which only a small corpus of material will be developed (10 hours of transcribed audio and 90 hours of untranscribed audio). The objectives of this project are both theoretical and substantive. For the first, this project will advance the development of natural language processing for low-resource languages and establish a protocol for extending this to other related languages. Substantively, this project will produce an unprecedented corpus of transcribed audio for five related languages, facilitating the comparative study of these languages by theoretical and descriptive linguists. State-of-the-art automatic speech recognition (ASR) depends upon the existence of a corpus of material (audio recordings with time-coded transcriptions) and the application of artificial intelligence systems that utilize neural networks to replicate humans learning by interpreting raw data. This present project employs what is called an "end-to-end neural network." Effectively, the artificial neural network is presented with input data (the acoustic speech signal) and a prepared the end result (a transcription) and learns to achieve the same result. To accomplish this, the original corpus is divided into training (~ 80%), validation (~10%), and test (~10%) sets. For endangered language documentation the goal is not simply accuracy of the ASR system but also the reduction of human effort to achieve highly accurate time-coded transcriptions that will be archived as a permanent record of target language. The project team has already developed a highly accurate system for one phonologically difficult tonal language (character error rate 8%) and reduced the human effort required to produce an accurate time-coded transcription by 75% (from 40 hours needed by a human starting from scratch to 9 hours needed by a human proofing a transcription generated by ASR). For this project the same team will explore ASR strategies for a morphologically complex agglutinative language in the hope of achieving the same degree of accuracy and reduction in human effort. This project will also address another challenge for state-of-the-art ASR: The transfer of an effective system developed for one language to low-resource, virtually undocumented related languages. Should the project be successful it will serve as a model for similar efforts with other languages and language groups. The data and findings will be available at Linguistic Data Consortium at the University of Pennsylvania, and Sam Noble Oklahoma Museum of Natural History, University of Oklahoma.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.