Detecting Failures of Neural Machine Translation in the Absence of Reference Translations
Detecting Failures of Neural Machine Translation in the Absence of Reference Translations
复制标题
在缺乏参考翻译的情况下检测神经机器翻译的故障
DOI：
10.1109/dsn-industry.2019.00007
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
Xie, Tao
中科院分区：
文献类型：
--
作者：
Wang, Wenyu;Zheng, Wujie;Liu, Dian;Zhang, Changrong;Zeng, Qinsong;Deng, Yuetang;Yang, Wei;He, Pinjia;Xie, Tao
尽管神经机器翻译 (NMT) 系统最近得到了广泛采用，但人们经常发现它会在输出中产生翻译失败。开发人员一直依靠内部系统测试来保证 NMT 的质量。这种测试方法需要人工构建的参考翻译作为基本事实（测试预言），例如自然语言输入。测试方法显示了在早期开发阶段快速增强 NMT 系统的好处。然而，在工业环境中，希望在不依赖参考翻译的情况下检测翻译失败，以便进一步提高工业开发和生产环境中的翻译质量。为了针对工业环境中的此类需求提供实用且可扩展的解决方案，在本文中，我们提出了一种自动识别翻译失败的新方法，而不需要翻译任务的参考翻译。我们的方法侧重于自然语言翻译的属性，可以通过使用来自 NMT 系统的测试输入（即要翻译的文本）和测试输出（即正在检查的翻译）的信息来系统地检查该属性。我们对现实世界数据集进行的评估表明，我们的方法可以有效地将财产违规检测为翻译失败。通过在微信（每月活跃用户超过 10 亿的即时通讯应用程序）的翻译服务中部署我们的方法，我们证明了我们的方法在工业环境中既实用又可扩展。
Despite getting widely adopted recently, a Neural Machine Translation (NMT) system is often found to produce translation failures in the outputs. Developers have been relying on in-house system testing for quality assurance of NMT. This testing methodology requires human-constructed reference translations as the ground truth (test oracle) for example natural language inputs. The testing methodology has shown benefits of quickly enhancing an NMT system in early development stages. However, in industrial settings, it is desirable to detect translation failures without reliance on reference translations for enabling further improvements on translation quality in both industrial development and production environments. Aiming for a practical and scalable solution to such demand in the industrial settings, in this paper, we propose a new approach for automatically identifying translation failures without requiring reference translations for a translation task. Our approach focuses on a property of natural language translation that can be checked systematically by using information from both the test inputs (i.e., the texts to be translated) and the test outputs (i.e., the translations under inspection) of the NMT system. Our evaluation conducted on real-world datasets shows that our approach can effectively detect property violations as translation failures. By deploying our approach in the translation service of WeChat (a messenger app with more than one billion monthly active users), we show that our approach is both practical and scalable in the industrial settings.
DOI：
10.1109/icse-companion.2019.00131
发表时间：
2018-07
期刊：
2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)
影响因子：
--
作者：
Wujie Zheng;Wenyu Wang;Dian Liu;Changrong Zhang;Qinsong Zeng;Yuetang Deng;Wei Yang;Pinjia He;Tao Xie
通讯作者：
Wujie Zheng;Wenyu Wang;Dian Liu;Changrong Zhang;Qinsong Zeng;Yuetang Deng;Wei Yang;Pinjia He;Tao Xie
使用体内测试方法保证软件应用程序的质量
DOI：
10.1109/icst.2009.18
发表时间：
2009
期刊：
2009 International Conference on Software Testing Verification and Validation
影响因子：
--
作者：
Christian Murphy;G. Kaiser;Ian Vo;Matt Chu
通讯作者：
Matt Chu