Distilling the Knowledge of BERT for Sequence-to-Sequence ASR
Distilling the Knowledge of BERT for Sequence-to-Sequence ASR
复制标题
DOI：
10.21437/interspeech.2020-1179
复制
发表时间：
2020-08
期刊：
影响因子：
--
通讯作者：
Hayato Futami;H. Inaguma;Sei Ueno;M. Mimura;S. Sakai;Tatsuya Kawahara
中科院分区：
文献类型：
--
作者：
Hayato Futami;H. Inaguma;Sei Ueno;M. Mimura;S. Sakai;Tatsuya Kawahara
基于注意力的序列到序列（seq2seq）模型在自动语音识别（ASR）领域取得了可喜的成果。然而，由于这些模型以从左到右的方式解码，因此它们无法访问右侧的上下文。我们通过知识蒸馏将 BERT 作为外部语言模型应用于 seq2seq ASR，从而利用左右上下文。在我们提出的方法中，BERT 生成软标签来指导 seq2seq ASR 的训练。此外，我们利用当前话语之外的上下文作为 BERT 的输入。实验评估表明，我们的方法在日语自发语料库 (CSJ) 上的 seq2seq 基线上显着提高了 ASR 性能。 BERT 的知识蒸馏优于仅查看左侧上下文的 Transformer LM。我们还展示了利用当前话语之外的上下文的有效性。我们的方法优于其他 LM 应用方法，例如 n 最佳重新评分和浅层融合，同时它不需要额外的推理成本。
Attention-based sequence-to-sequence (seq2seq) models have achieved promising results in automatic speech recognition (ASR). However, as these models decode in a left-to-right way, they do not have access to context on the right. We leverage both left and right context by applying BERT as an external language model to seq2seq ASR through knowledge distillation. In our proposed method, BERT generates soft labels to guide the training of seq2seq ASR. Furthermore, we leverage context beyond the current utterance as input to BERT. Experimental evaluations show that our method significantly improves the ASR performance from the seq2seq baseline on the Corpus of Spontaneous Japanese (CSJ). Knowledge distillation from BERT outperforms that from a transformer LM that only looks at left context. We also show the effectiveness of leveraging context beyond the current utterance. Our method outperforms other LM application approaches such as n-best rescoring and shallow fusion, while it does not require extra inference cost.