MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages
MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages
复制标题
DOI：
10.48550/arxiv.2203.08388
复制
发表时间：
2022-03
期刊：
影响因子：
--
通讯作者：
Zhiruo Wang;Grace Cuenca;Shuyan Zhou;Frank F. Xu;Graham Neubig
中科院分区：
文献类型：
--
作者：
Zhiruo Wang;Grace Cuenca;Shuyan Zhou;Frank F. Xu;Graham Neubig
虽然最近在天然和编程语言的交集（例如代码生成和代码摘要）的交集中，这些应用程序通常以英语为中心。这为不精通英语的计划开发人员造成了障碍。为了减轻跨语言技术发展的这一差距，我们提出了一个多语言数据集McOnala，以从自然语言命令中生成基准代码，从而扩展了英语。以英语代码/自然语言挑战（CONALA）数据集为基础，我们注释了三种语言的896个NL代码对：西班牙语，日语和俄语。我们通过测试最先进的代码生成系统来对McOnala进行系统评估。尽管三种语言的困难各不相同，但所有系统都显着落后于英语，这揭示了将代码生成对新语言的挑战。
While there has been a recent burgeoning of applications at the intersection of natural and programming languages, such as code generation and code summarization, these applications are usually English-centric. This creates a barrier for program developers who are not proficient in English. To mitigate this gap in technology development across languages, we propose a multilingual dataset, MCoNaLa, to benchmark code generation from natural language commands extending beyond English. Modeled off of the methodology from the English Code/Natural Language Challenge (CoNaLa) dataset, we annotated a total of 896 NL-Code pairs in three languages: Spanish, Japanese, and Russian. We present a systematic evaluation on MCoNaLa by testing state-of-the-art code generation systems. Although the difficulties vary across three languages, all systems lag significantly behind their English counterparts, revealing the challenges in adapting code generation to new languages.