Testing Untestable Neural Machine Translation: An Industrial Case
Testing Untestable Neural Machine Translation: An Industrial Case
复制标题
DOI：
10.1109/icse-companion.2019.00131
复制
发表时间：
2018-07
期刊：
影响因子：
--
通讯作者：
Wujie Zheng;Wenyu Wang;Dian Liu;Changrong Zhang;Qinsong Zeng;Yuetang Deng;Wei Yang;Pinjia He;Tao Xie
中科院分区：
文献类型：
--
作者：
Wujie Zheng;Wenyu Wang;Dian Liu;Changrong Zhang;Qinsong Zeng;Yuetang Deng;Wei Yang;Pinjia He;Tao Xie
神经机器翻译（NMT）表现出很大的优势，并且越来越受欢迎。但是，实际上，NMT经常在翻译中产生意外的翻译失败。虽然基于参考的黑盒系统测试一直是开发过程中NMT质量保证的普遍实践，但一种越来越重要的工业实践，称为Vivo In-Vivo测试，在实际用户使用已部署的工业NMT时，暴露了看不见的类型或翻译失败的实例系统。为了填补缺乏用于NMT系统体内测试的测试甲壳的空白，我们提出了一种新方法，用于自动识别不参考翻译的翻译失败。我们在现实世界数据集上进行的评估表明，我们的方法论有效地检测了几种针对性的翻译失败类型。我们在微信（具有超过10亿个活跃用户的Messenger应用程序）中部署方法的经验表明，我们的方法论很高的有效性以及行业的较高影响。
Neural Machine Translation (NMT) has shown great advantages and is becoming increasingly popular. However, in practice, NMT often produces unexpected translation failures in its translations. While reference-based black-box system testing has been a common practice for NMT quality assurance during development, an increasingly critical industrial practice, named in-vivo testing, exposes unseen types or instances of translation failures when real users are using a deployed industrial NMT system. To fill the gap of lacking test oracles for in-vivo testing of NMT systems, we propose a new methodology for automatically identifying translation failures without reference translations. Our evaluation conducted on real-world datasets shows that our methodology effectively detects several targeted types of translation failures. Our experiences on deploying our methodology in both production and development environments of WeChat (a messenger app with over one billion monthly active users) demonstrate high effectiveness of our methodology along with high industry impact.