Improved Prosody from Learned F0 Codebook Representations for VQ-VAE Speech Waveform Reconstruction
Improved Prosody from Learned F0 Codebook Representations for VQ-VAE Speech Waveform Reconstruction
复制标题
DOI：
10.21437/interspeech.2020-1615
复制
发表时间：
2020-05
期刊：
影响因子：
--
通讯作者：
Yi Zhao;Haoyu Li;Cheng-I Lai;Jennifer Williams;Erica Cooper;J. Yamagishi
中科院分区：
文献类型：
--
作者：
Yi Zhao;Haoyu Li;Cheng-I Lai;Jennifer Williams;Erica Cooper;J. Yamagishi
矢量量化变分自编码器（VQ-VAE）是一种强大的表征学习框架，可以在没有监督的情况下从语音信号中发现离散的特征组。到目前为止，VQ-VAE体系结构之前已经建模了个别类型的语音特征，例如仅针对电话或仅针对F0。本文介绍了VQ-VAE的一个重要扩展，用于同时学习与f0相关的超段信息以及传统的电话特征。所提出的框架使用两个编码器，使得F0轨迹和语音波形都是系统的输入，因此两个单独的码本被学习。我们使用WaveRNN声码器作为VQ-VAE的解码器组件。我们的独立于说话人的VQ-VAE使用来自多说话人日语语音数据库的原始语音波形进行训练。实验结果表明，所提出的扩展减少了所有未见过的测试说话者重构语音的F0失真，并显著提高了听力测试的偏好分数。我们还使用单语普通话语音进行了实验，以证明我们的架构在另一种严重依赖F0的语言中的优势。
Vector Quantized Variational AutoEncoders (VQ-VAE) are a powerful representation learning framework that can discover discrete groups of features from a speech signal without supervision. Until now, the VQ-VAE architecture has previously modeled individual types of speech features, such as only phones or only F0. This paper introduces an important extension to VQ-VAE for learning F0-related suprasegmental information simultaneously along with traditional phone features.The proposed framework uses two encoders such that the F0 trajectory and speech waveform are both input to the system, therefore two separate codebooks are learned. We used a WaveRNN vocoder as the decoder component of VQ-VAE. Our speaker-independent VQ-VAE was trained with raw speech waveforms from multi-speaker Japanese speech databases. Experimental results show that the proposed extension reduces F0 distortion of reconstructed speech for all unseen test speakers, and results in significantly higher preference scores from a listening test. We additionally conducted experiments using single-speaker Mandarin speech to demonstrate advantages of our architecture in another language which relies heavily on F0.