Study of Class-based Language Model and its Application to Japanese Morphological Analysis

词法分析是日语处理中最基本的过程。在日语词法分析中，分词是一个重要的问题，因为它的书写系统中没有标记词的边界。在本研究项目中，我们首先研究了一个基于字符的n元语法模型的分词模型，这是我们的基线方法。接下来，我们将PPM*压缩算法应用于分词问题。PPM(部分匹配预测)是一种基于有限上下文概率建模技术的无损压缩算法，PPM*是PPM的一种变体，它对上下文长度没有先验限制。字符类模型比基于字符的模型更健壮，因为字符类模型的参数数量少于基于字符的模型。日文字符聚类的度量是不同于模型估计的语料库上的熵，搜索方法是基于贪婪算法。因此，这种聚类方法在不给出类的数量的情况下给出了最优的字符分类。在ADD(ATR Dialog Database)语料库上的实验结果表明，本文提出的基于字类模型的日语分词方法比基于字符的分词方法具有更高的准确率。特别是，对于开放文本，使用可变长度n-gram类模型的方法获得了96.38%的召回率和96.23%的准确率。

Morphological analysis is the most fundamental process of Japanese language processing. In Japanese morphological analysis, word segmentation is an important problem because word boundaries are not marked in its writing system.In this research project, we first studied a word segmentation model using a character-based n-gram model, which is our baseline method. Next, we applied the PPM* compression algorithm to the problem of word segmentation. PPM (Prediction by Partial Matching) is a lossless compression algorithm based on a finite-context probabilistic modeling technique and PPM* is a variant of PPM, in which there is no a priori bound on context length.We then studied a method for word segmentation based on a character class model. The character class model is more robust than a character-based model because the number of parameters of the character class model is fewer than that of a character-based model. The measurement for Japanese character clustering is the entropy on a corpus different from the corpus for model estimation and the search method is based on the greedy algorithm. For this reason, this clustering method gives us an optimum character classification without giving the number of classes. As the result of experiments on the ADD (ATR Dialogue Database) corpus, the proposed Japanese word segmenter using the character class model marked a higher accuracy than a character-based model. In particular, the proposed method using a variable-length n;-gram class model achieved 96.38% recall and 96.23% precision for open text.