Unsupervised Segmentation of Chinese Text by Use of Branching Entropy
Unsupervised Segmentation of Chinese Text by Use of Branching Entropy
复制标题
DOI：
10.3115/1273073.1273129
复制
发表时间：
2006-07
期刊：
影响因子：
--
通讯作者：
Zhihui Jin;Kumiko Tanaka-Ishii
中科院分区：
文献类型：
--
作者：
Zhihui Jin;Kumiko Tanaka-Ishii
We propose an unsupervised segmentation method based on an assumption about language data: that the increasing point of entropy of successive characters is the location of a word boundary. A large-scale experiment was conducted by using 200 MB of unsegmented training data and 1 MB of test data, and precision of 90% was attained with recall being around 80%. Moreover, we found that the precision was stable at around 90% independently of the learning data size.
