CAREER: Modeling Spoken Language Without Parallel Text Annotations

自动语音识别和理解技术已被广泛应用于个人数字助理、视频和会议的自动转录以及更多应用中。建立这些系统需要大量的语音音频数据集，这些数据集被人类转录成文本。当特定领域有足够的数据可用时，基于深度神经网络的现代模型能够执行高精度的语音识别和下游语言理解任务。然而，对于世界上7000种语言中的绝大多数以及更多的方言来说，大规模的注释数据集根本不存在，阻碍了语音技术为这些语言及其使用者服务。受人类在会读或写之前很久就学会说话这一事实的启发，这个职业项目探索了一种不依赖于转录语音的语音处理新范式。相反，它开发了能够直接从语音音频学习口语的新模型，并将这些模型应用于包括构建无需转录语音的语音识别器以及将一种语言自动翻译成另一种语言的任务。这些进展符合研究界的一项更大的运动，以显著降低成本，并增加语音识别和理解技术的可用性，使更多的语言和用户能够使用。该项目利用自我监督和多模式学习方法来自动发现语言结构(电话、单词、短语等)在原始语音信号中，该原始语音信号可以被视为“伪文本”，并且用于替代用于下游任务的常规文本。它为基于注意力的语音分割开发了新的神经网络层，以分层的方式应用于发现多个抽象级别的语音单元。第二种新颖的技术涉及使用分割层向模型添加自预测层和训练目标，其中将捕捉类词结构的较高层尝试预测捕捉子词结构的较低层的标记化。通过这种方式，模型可以自动学习语音词典，该词典捕捉所发现的语音单元的不同层之间的组成关系。该项目将这些技术应用于三个下游应用程序，它们在语音领域的重要性正在稳步增长：无监督语音识别、无文本语音到语音翻译以及用于对话和图像字幕的无文本语音生成。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Automatic speech recognition and understanding technology has been widely adopted into personal digital assistants, automatic transcription of videos and meetings, and many more applications. Building these systems requires massive datasets of speech audio that is human-transcribed into text. When sufficient data is available for a particular domain, modern models based on deep neural networks are capable of highly accurate speech recognition and downstream language understanding tasks. However, for the vast majority of the world's 7,000 languages and even more numerous dialects, large scale annotated datasets simply do not exist, preventing speech technology from serving these languages and their speakers. Inspired by the fact that humans learn to speak long before they can read or write, this CAREER project explores a new paradigm for speech processing that does not rely on transcribed speech. Instead, it develops new models that are capable of learning spoken language directly from speech audio, and applies these models to tasks including building speech recognizers without transcribed speech and automatically translating speech from one language into another. These advances fit within a larger movement in the research community to dramatically reduce the cost and increase the availability of speech recognition and understanding technology to many more languages and users than are served today.This project leverages self-supervised and multimodal learning approaches to automatically discover linguistic structure (phones, words, phrases, etc.) in the raw speech signal which can be treated as ``pseudo-text'' and used in place of conventional text for downstream tasks. It develops new neural network layers for attention-based segmentation of speech, applied in a hierarchical fashion to discover speech units at multiple levels of abstraction. A second novel technique involves adding self-prediction layers and training objectives to a model using the segmentation layers, where the higher layers that would capture word-like structure attempt to predict the tokenization of lower layers that capture sub-word structure. In this way, the model can automatically learn a pronunciation lexicon that captures the compositional relationship between the different tiers of discovered speech units. The project applies these techniques to three downstream applications that are steadily growing in importance in the speech field: unsupervised speech recognition, textless speech-to-speech translation, and textless generation speech for dialog and image captioning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.