A Generalized Language Model as the Combination of Skipped n-grams and Modified Kneser Ney Smoothing
A Generalized Language Model as the Combination of Skipped n-grams and Modified Kneser Ney Smoothing
复制标题
跳过n元语法与改进Kneser Ney平滑相结合的广义语言模型
DOI：
10.3115/v1/p14-1108
复制
发表时间：
2014
期刊：
影响因子：
--
通讯作者：
Steffen Staab
中科院分区：
文献类型：
--
作者：
Rene Pickhardt;Thomas Gottron;Martin Körner;P. Wagner;Till Speicher;Steffen Staab
我们介绍了一种基于跳跃式n-gram模型的系统递归探索构建语言模型的新方法，该模型使用改进的Kneser-Ney平滑插值。我们的方法推广了语言模型，因为它包含了经典的低阶插值模型作为特例。在本文中，我们激励、形式化并提出了我们的方法。在对英语文本语料库进行的广泛的实证实验中，我们证明，与使用改进的Kneser-Ney平滑的传统语言模型相比，我们的广义语言模型导致困惑度大幅降低3.1%至12.7%。此外，我们研究了其他三种语言和特定领域语料库的行为，我们观察到一致的改进。最后，我们还表明，我们的方法的优势在于其处理稀疏训练数据的能力。使用一个只有736 KB文本的非常小的训练数据集，我们的困惑度甚至降低了25.7%。
We introduce a novel approach for building language models based on a systematic, recursive exploration of skip n-gram models which are interpolated using modified Kneser-Ney smoothing. Our approach generalizes language models as it contains the classical interpolation with lower order models as a special case. In this paper we motivate, formalize and present our approach. In an extensive empirical experiment over English text corpora we demonstrate that our generalized language models lead to a substantial reduction of perplexity between 3.1% and 12.7% in comparison to traditional language models using modified Kneser-Ney smoothing. Furthermore, we investigate the behaviour over three other languages and a domain specific corpus where we observed consistent improvements. Finally, we also show that the strength of our approach lies in its ability to cope in particular with sparse training data. Using a very small training data set of only 736 KB text we yield improvements of even 25.7% reduction of perplexity.