Optimizing Chinese Word Segmentation for Machine Translation Performance
Optimizing Chinese Word Segmentation for Machine Translation Performance
复制标题
DOI：
10.3115/1626394.1626430
复制
发表时间：
2008-06
期刊：
影响因子：
--
通讯作者：
Pi-Chuan Chang;Michel Galley;Christopher D. Manning
中科院分区：
文献类型：
--
作者：
Pi-Chuan Chang;Michel Galley;Christopher D. Manning
以前的研究表明，汉语分词对机器翻译是有用的，但不同的分词策略对机器翻译的影响仍然知之甚少。在本文中，我们证明了优化现有分割标准的分割并不总是产生更好的机器翻译性能。我们发现中文“词”的分割一致性和粒度等其他因素对于机器翻译更为重要。基于这些发现，我们在一个条件随机场分割器中实现了直接优化MT任务的分割粒度的方法，提供了0.73 BLEU的改进。我们还表明，使用外部词汇和专有名词特征来提高分词一致性，BLEU增加了0.32。
Previous work has shown that Chinese word segmentation is useful for machine translation to English, yet the way different segmentation strategies affect MT is still poorly understood. In this paper, we demonstrate that optimizing segmentation for an existing segmentation standard does not always yield better MT performance. We find that other factors such as segmentation consistency and granularity of Chinese "words" can be more important for machine translation. Based on these findings, we implement methods inside a conditional random field segmenter that directly optimize segmentation granularity with respect to the MT task, providing an improvement of 0.73 BLEU. We also show that improving segmentation consistency using external lexicon and proper noun features yields a 0.32 BLEU increase.