A study on optimazation of units for statistical language models

在这个项目中，我们调查并重新考虑了两种“单位”作为统计语言模型的基本属性。我们重新考虑的第一个单位是“符号”或“字典条目”，它们是句子的最小单位。普通的统计语言模型使用单词或字符作为标记。但是对于某些应用程序，如机器翻译，我们知道使用较长的标记（如短语）可以提高系统性能。我们专注于自动短语提取，以建立具有统计标准的机器翻译词典。提出了最小互信息标准，并证明了该方法优于以往的短语提取方法。我们重新考虑的另一种单位是由模型评估的“目标”。普通的统计语言模型评估“句子”作为应用的目标。但是许多语言应用程序必须输出由多个句子组成的文本。我们提出了一个用Dirichlet混合物作为多项式分布参数分布的模型来评估全文，其复合分布为Polya混合物。我们发现我们的模型比其他文本模型（如潜在狄利克雷分配（LDA））更低的困惑。对已读文档的语音识别实验表明，该模型利用全文信息有效地纠正了许多误读词。

In this project, we investigated and reconsidered two kinds of ‘units' as a basic property of statistical language models. The first unit we reconsidered is ‘tokens' or ‘entries of a dictionary' which are minimal units of sentences. Ordinary statistical language models use words or characters as tokens. But for some applications such as machine translations, we know uses of longer tokens such as phrases improve the system performance. We focused on automatic phrase extractions to build up dictionaries for machine translations with a statistical criterion. We proposed new criteria, minimal mutual information, and showed the method is better than previous phrase extraction methods.Another kind of unit we reconsidered is ‘targets' which are assessed by the models. Ordinary statistical language models evaluate ‘sentences' as targets of applications. But many language applications have to output text which is made up with multiple sentences. We proposed a model to evaluate whole text using Dirichlet mixtures as a distribution for parameters of a multinomial distribution, whose compound distribution is Polya mixtures. We showed lower perplexity of our model than that of the other text models such as the latent Dirichlet allocation(LDA). Experiments of speech recognizer for read documents showed the models effectively correct many misrecognition words using information of whole text.