Distilhubert: Speech Representation Learning by Layer-Wise Distillation of Hidden-Unit Bert
Distilhubert: Speech Representation Learning by Layer-Wise Distillation of Hidden-Unit Bert
复制标题
Distilhubert：通过隐藏单元 Bert 的分层蒸馏进行语音表示学习
DOI：
--
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Hung
中科院分区：
文献类型：
--
作者：
Heng;Shu;Hung
自监督语音表示学习方法，如Wav2vec 2.0和隐藏单元BERT(Hubert)，利用未标记的语音数据进行预训练，并为大量语音处理任务提供良好的表示。尽管这些方法取得了成功，但它们需要大容量内存和高昂的预培训成本，使得学术界和小公司的研究人员无法接触到它们。为此，本文介绍了一种新的多任务学习框架DistilHuBERT，用于直接从Hubert模型中提取隐含表示。这种方法将Hubert的规模减少了75%，速度提高了73%，同时在10个不同的任务中保持了大部分性能。此外，DistilHuBERT只需要很少的培训时间和数据，为语音的个人和设备上的SSL模型提供了预先培训的可能性。
Self-supervised speech representation learning methods like wav2vec 2.0 and Hidden-unit BERT (HuBERT) leverage unlabeled speech data for pre-training and offer good representations for numerous speech processing tasks. Despite the success of these methods, they require large memory and high pre-training costs, making them inaccessible for researchers in academia and small companies. Therefore, this paper introduces DistilHuBERT, a novel multi-task learning framework to distill hidden representations from a HuBERT model directly. This method reduces HuBERT’s size by 75% and 73% faster while retaining most performance in ten different tasks. Moreover, DistilHuBERT required little training time and data, opening the possibilities of pre-training personal and on-device SSL models for speech.
DOI：
10.1109/asru51503.2021.9688093
发表时间：
2021-07
期刊：
2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)
影响因子：
--
作者：
Ankita Pasad;Ju-Chieh Chou;Karen Livescu
通讯作者：
Ankita Pasad;Ju-Chieh Chou;Karen Livescu