Audio Lottery: Speech Recognition Made Ultra-Lightweight, Noise-Robust, and Transferable
Audio Lottery: Speech Recognition Made Ultra-Lightweight, Noise-Robust, and Transferable
复制标题
DOI：
--
复制
发表时间：
2022
期刊：
影响因子：
--
通讯作者：
Shaojin Ding;Tianlong Chen;Zhangyang Wang
中科院分区：
文献类型：
--
作者：
Shaojin Ding;Tianlong Chen;Zhangyang Wang
由于移动设备上越来越多的语音交互功能，轻量级语音识别模型出现了爆炸式的需求。由于从零开始设计这样的系统并非易事，从业者通常选择压缩大型（预训练的）语音模型。最近，彩票假设揭示了高度稀疏的子网络的存在，这些子网络可以在不牺牲完整模型性能的情况下进行孤立训练。在本文中，我们研究了使用彩票假设来发现轻量级语音识别模型的诱人可能性，这些模型：(1)对语音中存在的各种噪声具有鲁棒性；(2)可转移以适应开放世界的个性化；3)兼容结构化稀疏性。我们在CNN-LSTM、RNNTransducer和Transformer模型上进行了大量的实验，并验证了高度稀疏的“中奖票”的存在，这些中奖票可以在这些主干上匹配完整的模型性能。我们获得的中奖票在所有骨干上的权重不到全模型权重的20%，而最轻量级的只保留4.4%的权重。这些中奖的门票推广到结构化稀疏，没有性能损失，并且可以从大型源数据集异常地转移到各种目标数据集。也许最令人惊讶的是，当训练话语具有高背景噪声时，中奖票甚至大大优于完整模型，通过诱导稀疏性显示出噪声鲁棒性的额外奖励。代码可在https://github.com/VITA-Group/Audio-Lottery上获得。
Lightweight speech recognition models have seen explosive demands owing to a growing amount of speech-interactive features on mobile devices. Since designing such systems from scratch is non-trivial, practitioners typically choose to compress large (pre-trained) speech models. Recently, lottery ticket hypothesis reveals the existence of highly sparse subnetworks that can be trained in isolation without sacrificing the performance of the full models. In this paper, we investigate the tantalizing possibility of using lottery ticket hypothesis to discover lightweight speech recognition models, that are (1) robust to various noise existing in speech; (2) transferable to fit the open-world personalization; and 3) compatible with structured sparsity. We conducted extensive experiments on CNN-LSTM, RNNTransducer, and Transformer models, and verified the existence of highly sparse “winning tickets” that can match the full model performance across those backbones. We obtained winning tickets that have less than 20% of full model weights on all backbones, while the most lightweight one only keeps 4.4% weights. Those winning tickets generalize to structured sparsity with no performance loss, and transfer exceptionally from large source datasets to various target datasets. Perhaps most surprisingly, when the training utterances have high background noises, the winning tickets even substantially outperform the full models, showing the extra bonus of noise robustness by inducing sparsity. Codes are available at https://github.com/VITA-Group/Audio-Lottery.