Study of Class-based Language Model and its Application to Japanese Morphological Analysis

词形分析是日语语言处理中最基本的过程。在日语的词法分析中，分词是一个重要的问题，因为它的书写系统中没有标记词的边界。在这个研究项目中，我们首先使用基于字符的n-gram模型研究了一个分词模型，这是我们的基线方法。接下来，我们将PPM*压缩算法应用于分词问题。PPM （Prediction by Partial Matching）是一种基于有限上下文概率建模技术的无损压缩算法，PPM*是PPM的一种变体，其中对上下文长度没有先验限制。然后，我们研究了一种基于字符类模型的分词方法。字符类模型比基于字符的模型更健壮，因为字符类模型的参数数量比基于字符的模型少。日文字符聚类的度量是不同于模型估计的语料库上的熵，搜索方法是基于贪心算法。出于这个原因，这种聚类方法在不给出类别数量的情况下为我们提供了最佳的字符分类。在ADD （ATR Dialogue Database）语料库上的实验结果表明，采用字符类模型的日语分词方法比基于字符的分词方法准确率更高。特别地，所提出的方法采用变长n；-gram类模型对开放文本的查全率为96.38%，查准率为96.23%。

Morphological analysis is the most fundamental process of Japanese language processing. In Japanese morphological analysis, word segmentation is an important problem because word boundaries are not marked in its writing system.In this research project, we first studied a word segmentation model using a character-based n-gram model, which is our baseline method. Next, we applied the PPM* compression algorithm to the problem of word segmentation. PPM (Prediction by Partial Matching) is a lossless compression algorithm based on a finite-context probabilistic modeling technique and PPM* is a variant of PPM, in which there is no a priori bound on context length.We then studied a method for word segmentation based on a character class model. The character class model is more robust than a character-based model because the number of parameters of the character class model is fewer than that of a character-based model. The measurement for Japanese character clustering is the entropy on a corpus different from the corpus for model estimation and the search method is based on the greedy algorithm. For this reason, this clustering method gives us an optimum character classification without giving the number of classes. As the result of experiments on the ADD (ATR Dialogue Database) corpus, the proposed Japanese word segmenter using the character class model marked a higher accuracy than a character-based model. In particular, the proposed method using a variable-length n;-gram class model achieved 96.38% recall and 96.23% precision for open text.