Benchmarking Language Models for Code Syntax Understanding
Benchmarking Language Models for Code Syntax Understanding
复制标题
DOI：
10.48550/arxiv.2210.14473
复制
发表时间：
2022-10
期刊：
影响因子：
--
通讯作者：
Da Shen;Xinyun Chen;Chenguang Wang;Koushik Sen;Dawn Song
中科院分区：
文献类型：
--
作者：
Da Shen;Xinyun Chen;Chenguang Wang;Koushik Sen;Dawn Song
预先训练的语言模型在自然语言处理和程序理解方面都表现出了令人印象深刻的性能，它将输入表示为令牌序列，而无需显式地对其结构进行建模。一些先前的工作表明，预先训练的语言模型可以捕获自然语言的语法规则，而无需对语法理解任务进行微调。然而，到目前为止，对预训练模型如何理解代码结构的理解有限。在这项工作中，我们对最先进的预训练模型进行了第一次彻底的基准测试，用于识别程序的语法结构。具体来说，我们介绍Codebooks，一个大规模的数据集的程序注释的语法关系，在其相应的抽象语法树。我们的主要观察是，现有的语言模型在代码上进行了预训练，仍然缺乏对代码语法的理解。事实上，这些预先训练的编程语言模型无法匹配基于位置偏移和关键字的简单基线的性能。我们还提出了一个自然语言的基准突出自然语言和编程语言之间的差异，在语法结构的理解。我们的研究结果指出了现有的编程语言预训练方法的主要局限性，并提出了建模代码语法结构的重要性。
Pre-trained language models have demonstrated impressive performance in both natural language processing and program understanding, which represent the input as a token sequence without explicitly modeling its structure. Some prior works show that pre-trained language models can capture the syntactic rules of natural languages without finetuning on syntax understanding tasks. However, there is limited understanding of how well pre-trained models understand the code structure so far. In this work, we perform the first thorough benchmarking of the state-of-the-art pre-trained models for identifying the syntactic structures of programs. Specifically, we introduce CodeSyntax, a large-scale dataset of programs annotated with the syntactic relationships in their corresponding abstract syntax trees. Our key observation is that existing language models pretrained on code still lack the understanding of code syntax. In fact, these pre-trained programming language models fail to match the performance of simple baselines based on positional offsets and keywords. We also present a natural language benchmark to highlight the differences between natural languages and programming languages in terms of syntactic structure understanding. Our findings point out key limitations of existing pre-training methods for programming languages, and suggest the importance of modeling code syntactic structures.