Deep structured speech models

深层结构化语音模型

• 批准号: RGPIN-2021-02652
• 负责人: Boulianne, Gilles
• 金额: $ 2.77万
• 依托单位: Centre de recherche informatique de Montréal
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=764483
• 关键词: Deep; structured; speech; models

State-of-the-art speech recognition models are either end-to-end models or hybrid models. End-to-end models are entirely based on deep neural networks (DNN). Hybrid models combine an underlying structure of weighted finite-state automata (WFSA) with a surface layer of deep neural networks. When trained on large enough amounts of annotated recordings similar to the test data, end-to-end models can outperform hybrid models. For mismatched or smaller training data, hybrid models are often a better choice since they can use prior linguistic knowledge encoded in their structure to better generalize and avoid overfitting. However, in hybrid models, finite-state automata and deep neural components are not integrated; they are trained independently with different objectives and algorithms. As a result, a performance gap remains even when enough training data is available. In practice, hybrid models are complicated to train, require specialized coding, and cannot be easily integrated with common deep learning frameworks such as Pytorch or Tensorflow. Thus their implementations lag behind the latest developments in general deep learning. Recent proposals for differentiable automata suggest how they could be integrated with deep neural networks into a single model with an end-to-end differentiable loss, in a way that is efficient in time and space to be scalable enough for speech problems. This opens up several areas of research which are yet almost unexplored for speech modelling. I propose to work on three promising lines of investigation. 1- New architectures with joint training of WFSA and DNN parameters may bridge the performance gap when enough training data is available. 2 - New loss functions closer to actual sequence-based objective functions such as word or phoneme error rate should yield better performance than approximate losses used in deep neural only models. 3- Partial supervision afforded by structured generative models can significantly reduce the need for transcribed data. Although applicable to a wide range of problems, integrated models will have their largest impact where underlying structure is complex and annotated data is scarce. Thus I intend to apply them first to problems I encountered in my recent work on Indigenous languages spoken in Canada, ranging from subword analysis to speech recognition. Making speech technology accessible to these languages will benefit their transcription, preservation, and revitalization. This research addresses key limitations of current deep learning models in speech recognition, but potentially has broader applications in natural language processing, machine translation, or genomics, where sequence-to-sequence and segmentation problems are common. Because it combines the solid mathematical framework of probabilistic models with the practical, scalable methods of deep learning, this approach is well positioned to generate advances in knowledge while providing a rich learning environment.

最先进的语音识别模型要么是端到端模型，要么是混合模型。端到端模型完全基于深度神经网络(DNN)。混合模型结合了加权有限状态自动机(WFSA)的底层结构和深层神经网络的表层。当在足够多的类似于测试数据的带注释的录音上进行训练时，端到端模型可以优于混合模型。对于不匹配或较小的训练数据，混合模型通常是更好的选择，因为它们可以使用编码在其结构中的先前语言知识来更好地泛化和避免过拟合。然而，在混合模型中，有限状态自动机和深层神经组件并不是集成在一起的，它们是用不同的目标和算法独立训练的。因此，即使有足够的训练数据可用，性能差距仍然存在。在实践中，混合模型训练起来很复杂，需要专门的编码，而且不能很容易地与常见的深度学习框架(如Pytorch或TensorFlow)集成。因此，它们的实现落后于一般深度学习的最新发展。最近关于可微自动机的提议表明，它们如何与深度神经网络集成到一个具有端到端可微损失的单一模型中，以一种在时间和空间上高效的方式来足够可扩展地处理语音问题。这打开了几个研究领域，这些领域在语音建模方面几乎还没有被探索过。我建议致力于三条有希望的调查路线。1-当有足够的训练数据时，具有WFSA和DNN参数的联合训练的新体系结构可能会弥补性能差距。2-新的损失函数更接近实际的基于序列的目标函数，例如单词或音素错误率，应该比深度神经模型中使用的近似损失产生更好的性能。结构化生成模型提供的部分监督可以显著减少对转录数据的需求。尽管集成模型适用于广泛的问题，但在底层结构复杂和注释数据稀缺的情况下，集成模型将产生最大影响。因此，我打算首先将它们应用到我最近关于加拿大土著语言的工作中遇到的问题，从子词分析到语音识别。使这些语言能够使用语音技术将有利于它们的转录、保存和振兴。这项研究解决了当前深度学习模型在语音识别中的关键局限性，但在自然语言处理、机器翻译或基因组学中具有更广泛的应用前景，在这些领域，序列到序列和分段问题很常见。由于它结合了可靠的概率模型的数学框架和实用的、可扩展的深度学习方法，这种方法处于有利地位，可以在提供丰富的学习环境的同时产生知识进步。