InferCode: Self-Supervised Learning of Code Representations by Predicting Subtrees
InferCode: Self-Supervised Learning of Code Representations by Predicting Subtrees
复制标题
DOI：
10.1109/icse43902.2021.00109
复制
发表时间：
2020-12
期刊：
影响因子：
--
通讯作者：
Nghi D. Q. Bui;Yijun Yu;Lingxiao Jiang
中科院分区：
文献类型：
--
作者：
Nghi D. Q. Bui;Yijun Yu;Lingxiao Jiang
学习代码表示在软件工程中有许多用途，例如代码分类，代码搜索，注释生成和错误预测等。尽管已经提出了令牌，语法树，依赖图，树中的路径或其变体的组合中的代码表示，但现有的学习技术具有主要限制，即这些模型通常在标记为特定下游任务的数据集上训练，因此代码表示可能不适合于其它任务。即使一些技术从未标记的代码生成表示，当应用于下游任务时，它们也远不能令人满意。为了克服这一局限性，本文提出了InferCode，它将自然语言处理中的自监督学习思想适应于代码的抽象语法树（AST）。新颖之处在于通过预测从AST的上下文自动识别的子树来训练代码表示。使用InferCode，AST中的子树被视为用于训练代码表示的标签，而无需任何人工标记工作或昂贵的图形构造开销，并且训练的表示不再与任何特定的下游任务或代码单元绑定。我们已经使用基于树的卷积神经网络（TBCNN）训练了一个InferCode模型的实例，作为一个大型Java代码集的编码器。然后，这个预训练的模型可以应用于下游的无监督任务，如代码聚类、代码克隆检测、跨语言代码搜索，或者在迁移学习方案下重复使用，以继续训练有监督任务的模型权重，如代码分类和方法名称预测。与应用于相同下游任务的现有技术（如code2vec，code2seq，ASTNN）相比，使用我们预先训练的InferCode模型，大多数任务（包括涉及不同编程语言的任务）都可以实现更高的性能。InferCode的实现和经过训练的嵌入可在链接https://github.com/bdqnghi/infercode上获得。
Learning code representations has found many uses in software engineering, such as code classification, code search, comment generation, and bug prediction, etc. Although representations of code in tokens, syntax trees, dependency graphs, paths in trees, or the combinations of their variants have been proposed, existing learning techniques have a major limitation that these models are often trained on datasets labeled for specific downstream tasks, and as such the code representations may not be suitable for other tasks. Even though some techniques generate representations from unlabeled code, they are far from being satisfactory when applied to the downstream tasks. To overcome the limitation, this paper proposes InferCode, which adapts the self-supervised learning idea from natural language processing to the abstract syntax trees (ASTs) of code. The novelty lies in the training of code representations by predicting subtrees automatically identified from the contexts of ASTs. With InferCode, subtrees in ASTs are treated as the labels for training the code representations without any human labelling effort or the overhead of expensive graph construction, and the trained representations are no longer tied to any specific downstream tasks or code units. We have trained an instance of InferCode model using Tree-Based Convolutional Neural Network (TBCNN) as the encoder of a large set of Java code. This pre-trained model can then be applied to downstream unsupervised tasks such as code clustering, code clone detection, cross-language code search, or be reused under a transfer learning scheme to continue training the model weights for supervised tasks such as code classification and method name prediction. Compared to prior techniques applied to the same downstream tasks, such as code2vec, code2seq, ASTNN, using our pre-trained InferCode model higher performance is achieved with a significant margin for most of the tasks, including those involving different programming languages. The implementation of InferCode and the trained embeddings are available at the link: https://github.com/bdqnghi/infercode.