An Empirical Study on the Usage of BERT Models for Code Completion
An Empirical Study on the Usage of BERT Models for Code Completion
复制标题
DOI：
10.1109/msr52588.2021.00024
复制
发表时间：
2021-03
期刊：
影响因子：
--
通讯作者：
Matteo Ciniselli;Nathan Cooper;L. Pascarella;D. Poshyvanyk;M. D. Penta;G. Bavota
中科院分区：
文献类型：
--
作者：
Matteo Ciniselli;Nathan Cooper;L. Pascarella;D. Poshyvanyk;M. D. Penta;G. Bavota
代码完成是现代综合开发环境的主要功能之一（IDES）是通过预测开发人员在此领域进行研究的下一个代码代码来加快代码写作的。但是，这些技术的性能仍然仅限于在这项工作中进行的几个代币的预测。在支持不同粒度级别的代码完成中，最先进的深度学习模型（DL）模型，包括单个令牌，一个或多个整个语句，最多为整个代码块（例如，a for loop的迭代块）。为此，我们训练并测试了最近提出的罗伯塔模型的几种适应性变体，并从几个角度评估了其预测，包括：（i）在评估DL通用模型时通常采用的指标（即BLEU分数和Levenshtein距离）； ii）完美的预测百分比（即，与开发人员所写的预测代码段相匹配）；模型代表了代码完成的可行解决方案，在要求模型猜测整个块（最高约58％）的完美预测范围为〜7％，在从同一代码语句中掩盖的几个令牌的更简单方案中达到了〜58％。
Code completion is one of the main features of modern Integrated Development Environments (IDEs). Its objective is to speed up code writing by predicting the next code token(s) the developer is likely to write. Research in this area has substantially bolstered the predictive performance of these techniques. However, the support to developers is still limited to the prediction of the next few tokens to type. In this work, we take a step further in this direction by presenting a large-scale empirical study aimed at exploring the capabilities of state-of-the-art deep learning (DL) models in supporting code completion at different granularity levels, including single tokens, one or multiple entire statements, up to entire code blocks (e.g., the iterated block of a for loop). To this aim, we train and test several adapted variants of the recently proposed RoBERTa model, and evaluate its predictions from several perspectives, including: (i) metrics usually adopted when assessing DL generative models (i.e., BLEU score and Levenshtein distance); (ii) the percentage of perfect predictions (i.e., the predicted code snippets that match those written by developers); and (iii) the "semantic" equivalence of the generated code as compared to the one written by developers. The achieved results show that BERT models represent a viable solution for code completion, with perfect predictions ranging from ∼7%, obtained when asking the model to guess entire blocks, up to ∼58%, reached in the simpler scenario of few tokens masked from the same code statement.