Chinese Word Segmentation without Using Lexicon and Hand-crafted Training Data
Chinese Word Segmentation without Using Lexicon and Hand-crafted Training Data
复制标题
DOI：
10.3115/980691.980775
复制
发表时间：
1998-08
期刊：
影响因子：
--
通讯作者：
Maosong Sun;Dayang Shen;Benjamin Ka-Yin T'sou
中科院分区：
文献类型：
--
作者：
Maosong Sun;Dayang Shen;Benjamin Ka-Yin T'sou
Chinese word segmentation is the first step in any Chinese NLP system. This paper presents a new algorithm for segmenting Chinese texts without making use of any lexicon and hand-crafted linguistic resource. The statistical data required by the algorithm, that is, mutual information and the difference of t-score between characters, is derived automatically from raw Chinese corpora. The preliminary experiment shows that the segmentation accuracy of our algorithm is acceptable. We hope the gaining of this approach will be beneficial to improving the performance (especially in ability to cope with unknown words and ability to adapt to various domains) of the existing segmenters, though the algorithm itself can also be utilized as a stand-alone segmenter in some NLP applications.
