CoditT5: Pretraining for Source Code and Natural Language Editing
CoditT5: Pretraining for Source Code and Natural Language Editing
复制标题
DOI：
10.1145/3551349.3556955
复制
发表时间：
2022-08
期刊：
影响因子：
--
通讯作者：
Jiyang Zhang;Sheena Panthaplackel;Pengyu Nie;Junyi Jessy Li;Miloš Gligorić
中科院分区：
文献类型：
--
作者：
Jiyang Zhang;Sheena Panthaplackel;Pengyu Nie;Junyi Jessy Li;Miloš Gligorić
预训练语言模型已被证明在许多与软件相关的生成任务中是有效的；然而，它们并不适合编辑任务，因为它们不是设计来对编辑进行推理的。为了解决这个问题，我们提出了一个新的预训练目标，它明确地对编辑进行建模，并使用它来构建CoditT5， CoditT5是一个针对软件相关编辑任务的大型语言模型，它在大量源代码和自然语言注释上进行预训练。我们在各种下游编辑任务上对其进行微调，包括注释更新、错误修复和自动代码审查。通过优于标准的基于生成的模型，我们证明了我们的方法的通用性及其对编辑任务的适用性。我们还展示了标准生成模型和基于编辑的模型如何通过简单的重新排序策略相互补充，通过这种策略，我们为三个下游编辑任务实现了最先进的性能。
Pretrained language models have been shown to be effective in many software-related generation tasks; however, they are not well-suited for editing tasks as they are not designed to reason about edits. To address this, we propose a novel pretraining objective which explicitly models edits and use it to build CoditT5, a large language model for software-related editing tasks that is pretrained on large amounts of source code and natural language comments. We fine-tune it on various downstream editing tasks, including comment updating, bug fixing, and automated code review. By outperforming standard generation-based models, we demonstrate the generalizability of our approach and its suitability for editing tasks. We also show how a standard generation model and our edit-based model can complement one another through simple reranking strategies, with which we achieve state-of-the-art performance for the three downstream editing tasks.