Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining
Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining
复制标题
通过自监督语音和语言模型预训练进行半监督口语理解
DOI：
10.1109/icassp39728.2021.9414922
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
James R. Glass
中科院分区：
文献类型：
--
作者：
Cheng;Yung;Hung;Shang;James R. Glass
最近在口语理解(SLU)方面的许多工作至少在三个方面中的一个方面受到限制：模型被训练成关于甲骨文输入而忽略了ASR错误，模型被训练成只预测意图而没有时隙值，或者模型被训练基于大量的内部数据。在本文中，我们提出了一个干净而通用的框架，直接从语音中学习语义，并从转录或未转录的语音中进行半监督来解决这些问题。我们的框架建立在预先训练的端到端(E2E)ASR和自我监督的语言模型(如BERT)上，并在有限数量的目标SLU数据上进行了微调。我们研究了ASR部分的两种半监督设置：有监督的转录语音预训练，以及通过用自监督语音表示(如Wav2vec)替换ASR编码器的无监督预训练。同时，我们确定了评估SLU模型的两个基本标准：环境噪声稳健性和E2E语义评估。在ATIS上的实验表明，在存在环境噪声和可供训练的标注语义数据有限的情况下，以语音为输入的SLU框架在语义理解方面的性能与以Oracle文本为输入的SLU框架相当。
Much recent work on Spoken Language Understanding (SLU) is limited in at least one of three ways: models were trained on oracle text input and neglected ASR errors, models were trained to predict only intents without the slot values, or models were trained on a large amount of in-house data. In this paper, we propose a clean and general framework to learn semantics directly from speech with semi-supervision from transcribed or untranscribed speech to address these issues. Our framework is built upon pretrained end-to-end (E2E) ASR and self-supervised language models, such as BERT, and fine-tuned on a limited amount of target SLU data. We study two semi-supervised settings for the ASR component: supervised pretraining on transcribed speech, and unsupervised pretraining by replacing the ASR encoder with self-supervised speech representations, such as wav2vec. In parallel, we identify two essential criteria for evaluating SLU models: environmental noise-robustness and E2E semantics evaluation. Experiments on ATIS show that our SLU framework with speech as input can perform on par with those using oracle text as input in semantics understanding, even though environmental noise is present and a limited amount of labeled semantics data is available for training.