Scaling Hidden Markov Language Models
Scaling Hidden Markov Language Models
复制标题
DOI：
10.18653/v1/2020.emnlp-main.103
复制
发表时间：
2020-11
期刊：
影响因子：
--
通讯作者：
Justin T Chiu;Alexander M. Rush
中科院分区：
文献类型：
--
作者：
Justin T Chiu;Alexander M. Rush
隐马尔可夫模型(HMM)是用于序列建模的基本工具，它将隐藏状态与发射结构清楚地分开。然而，这种分离使得HMM很难适应现代自然语言处理中的大数据集，并且由于与完全观察的模型相比性能非常差而被淘汰。这项工作回顾了将HMM扩展到语言建模数据集的挑战，从最近的神经建模方法中吸取了一些想法。我们提出了将隐马尔可夫模型扩展到大规模状态空间的方法，同时保持有效的精确推理、紧凑的参数化和有效的正则化。实验表明，该方法得到的模型比以前的基于隐马尔可夫模型和n元语法的方法更准确，朝着最先进的神经模型的性能迈进了一步。
The hidden Markov model (HMM) is a fundamental tool for sequence modeling that cleanly separates the hidden state from the emission structure. However, this separation makes it difficult to fit HMMs to large datasets in modern NLP, and they have fallen out of use due to very poor performance compared to fully observed models. This work revisits the challenge of scaling HMMs to language modeling datasets, taking ideas from recent approaches to neural modeling. We propose methods for scaling HMMs to massive state spaces while maintaining efficient exact inference, a compact parameterization, and effective regularization. Experiments show that this approach leads to models that are more accurate than previous HMM and n-gram-based methods, making progress towards the performance of state-of-the-art neural models.