Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech
复制标题
从视觉基础语音中学习分层离散语言单元
DOI：
--
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
James R. Glass
中科院分区：
文献类型：
--
作者：
David F. Harwath;Wei;James R. Glass
在本文中，我们提出了一种通过在视觉基础语音的神经模型中加入矢量量化层来学习离散语言单位的方法。我们表明，我们的方法能够捕获单词级和子词单元，这取决于它是如何配置的。本文与以往关于语音单元学习的工作的不同之处在于训练目标的选择。我们不使用基于重建的损失，而是使用区分的、多模式的定位目标，迫使学习的单元对语义图像检索有用。我们评估了ZeroSpeech 2019挑战赛上的子字单元，在保持比特率大致不变的情况下，ABX错误率比表现最好的提交降低了27.3%。我们还展示了这些单元的噪声稳健性的实验。最后，我们证明了具有多个量化器的模型可以同时学习较低层的类电话检测器和较高层的类词检测器。我们表明，这些检测器是高精度的，发现了279个F1分数大于0.5的单词。
In this paper, we present a method for learning discrete linguistic units by incorporating vector quantization layers into neural models of visually grounded speech. We show that our method is capable of capturing both word-level and sub-word units, depending on how it is configured. What differentiates this paper from prior work on speech unit learning is the choice of training objective. Rather than using a reconstruction-based loss, we use a discriminative, multimodal grounding objective which forces the learned units to be useful for semantic image retrieval. We evaluate the sub-word units on the ZeroSpeech 2019 challenge, achieving a 27.3% reduction in ABX error rate over the top-performing submission, while keeping the bitrate approximately the same. We also present experiments demonstrating the noise robustness of these units. Finally, we show that a model with multiple quantizers can simultaneously learn phone-like detectors at a lower layer and word-like detectors at a higher layer. We show that these detectors are highly accurate, discovering 279 words with an F1 score of greater than 0.5.
DOI：
10.1109/taslp.2016.2517567
发表时间：
2016-03
期刊：
IEEE/ACM Transactions on Audio, Speech, and Language Processing
影响因子：
--
作者：
H. Kamper;A. Jansen;S. Goldwater
通讯作者：
H. Kamper;A. Jansen;S. Goldwater