An extensive study on pre-trained models for program understanding and generation
An extensive study on pre-trained models for program understanding and generation
复制标题
DOI：
10.1145/3533767.3534390
复制
发表时间：
2022-07
期刊：
影响因子：
--
通讯作者：
Zhengran Zeng;Hanzhuo Tan;Haotian Zhang;Jing Li;Yuqun Zhang;Lingming Zhang
中科院分区：
文献类型：
--
作者：
Zhengran Zeng;Hanzhuo Tan;Haotian Zhang;Jing Li;Yuqun Zhang;Lingming Zhang
自动程序理解和生成技术可以显着提高程序员的生产力，并已被学术界和工业界广泛研究。最近，预训练范式的出现启发研究人员开发通用的预训练模型，该模型可应用于广泛的程序理解和生成任务。这种预先训练的模型是通过大型未标记语料库的自监督目标得出的，可以在下游任务（例如代码搜索和代码生成）中以最小的适应进行微调。尽管这些预训练模型声称优于现有技术，但它们很少遵循等效的评估协议，例如，它们很难在相同的基准、任务或设置上进行评估。因此，迫切需要对预训练模型的有效性、通用性以及局限性进行全面研究，以便为该领域的未来发展提供启示和指导。为此，我们首先在七个代表性代码任务的大型基准上对八个开放访问预训练模型进行了广泛的研究，以评估其可重复性。我们进一步比较预训练模型和特定领域的最先进技术，以验证预训练的有效性。最后，我们通过检查预训练模型在对抗性攻击下的性能变化来研究预训练模型的鲁棒性。通过研究，我们发现，虽然我们通常可以复制预训练模型在其评估任务和采用的基准上的原始性能，但细微的性能波动可以反驳其原始论文中的发现。此外，现有的预训练模型都无法压倒所有其他模型。我们还发现，在程序理解任务中，预训练的模型可以显着优于非预训练的最先进技术。此外，我们通过对抗性攻击对自然语言编程语言预训练模型的鲁棒性进行了首次研究，发现简单的随机攻击方法可以轻松欺骗最先进的预训练模型，从而引发安全问题。最后，我们还提供了多种实用指南，以推进未来对用于程序理解和生成的预训练模型的研究。
Automatic program understanding and generation techniques could significantly advance the productivity of programmers and have been widely studied by academia and industry. Recently, the advent of pre-trained paradigm enlightens researchers to develop general-purpose pre-trained models which can be applied for a broad range of program understanding and generation tasks. Such pre-trained models, derived by self-supervised objectives on large unlabelled corpora, can be fine-tuned in downstream tasks (such as code search and code generation) with minimal adaptations. Although these pre-trained models claim superiority over the prior techniques, they seldom follow equivalent evaluation protocols, e.g., they are hardly evaluated on the identical benchmarks, tasks, or settings. Consequently, there is a pressing need for a comprehensive study of the pre-trained models on their effectiveness, versatility as well as the limitations to provide implications and guidance for the future development in this area. To this end, we first perform an extensive study of eight open-access pre-trained models over a large benchmark on seven representative code tasks to assess their reproducibility. We further compare the pre-trained models and domain-specific state-of-the-art techniques for validating pre-trained effectiveness. At last, we investigate the robustness of the pre-trained models by inspecting their performance variations under adversarial attacks. Through the study, we find that while we can in general replicate the original performance of the pre-trained models on their evaluated tasks and adopted benchmarks, subtle performance fluctuations can refute the findings in their original papers. Moreover, none of the existing pre-trained models can dominate over all other models. We also find that the pre-trained models can significantly outperform non-pre-trained state-of-the-art techniques in program understanding tasks. Furthermore, we perform the first study for natural language-programming language pre-trained model robustness via adversarial attacks and find that a simple random attack approach can easily fool the state-of-the-art pre-trained models and thus incur security issues. At last, we also provide multiple practical guidelines for advancing future research on pre-trained models for program understanding and generation.