It’s Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information
It’s Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information
复制标题
DOI：
10.18653/v1/2020.acl-main.149
复制
发表时间：
2020-05
期刊：
影响因子：
--
通讯作者：
Emanuele Bugliarello;Sabrina J. Mielke;Antonios Anastasopoulos;Ryan Cotterell;Naoaki Okazaki
中科院分区：
文献类型：
--
作者：
Emanuele Bugliarello;Sabrina J. Mielke;Antonios Anastasopoulos;Ryan Cotterell;Naoaki Okazaki
神经机器翻译系统的性能通常根据BLEU进行评估。然而，由于其依赖于目标语言属性和生成，BLEU度量不允许评估哪些翻译方向更难以建模。在本文中，我们提出了交叉互信息（XMI）：一种机器翻译难度的非对称信息理论度量，它利用了大多数神经机器翻译模型的概率性质。XMI允许我们更好地评估将文本翻译成目标语言的难度，同时控制独立于翻译任务的目标端生成组件的难度。然后，我们提出了第一个系统和控制的跨语言翻译困难的研究，使用现代神经翻译系统。复制我们实验的代码可以在https://github.com/e-bug/nmt-difficulty上找到。
The performance of neural machine translation systems is commonly evaluated in terms of BLEU. However, due to its reliance on target language properties and generation, the BLEU metric does not allow an assessment of which translation directions are more difficult to model. In this paper, we propose cross-mutual information (XMI): an asymmetric information-theoretic metric of machine translation difficulty that exploits the probabilistic nature of most neural machine translation models. XMI allows us to better evaluate the difficulty of translating text into the target language while controlling for the difficulty of the target-side generation component independent of the translation task. We then present the first systematic and controlled study of cross-lingual translation difficulties using modern neural translation systems. Code for replicating our experiments is available online at https://github.com/e-bug/nmt-difficulty.