SENLP - Software Engineering knowledge of NLP models

转换器体系结构改变了自然语言处理(NLP)领域，并为BERT和GPT等模型铺平了道路。这些模型的共同点是，它们以预训练的形式使用迁移学习来学习语言的一般表征，然后可以对其进行微调或提示执行各种下游任务。虽然这些模型在各种NLP任务中取得了显著的效果，但通常不清楚它们为什么在特定任务中表现良好，以及它们在不同领域(如软件工程(SE))中的工作情况。在我们之前的工作中，我们研究了特定领域的预培训对SE领域内NLP任务的影响，我们发现，对于“bug”(昆虫对缺陷)或“Root”(植物与用户)等多义词，特定领域的预培训确实有助于理解SE领域中的含义，这也有助于在特定于领域的下游任务中获得更好的表现。在这个项目中，我们希望加深对NLP模型从SE领域捕获概念的能力的理解，重点放在SE定义和常识知识上。我们将使用NLP模型作为学生的类比，以了解他们在SE考试中的表现。例如，我们将测试NLP模型是否包含准确的SE定义和术语：NLP模型能否在多项选择测试中发现术语的正确定义，它们能否在提示下生成准确的定义，它们是否能够理解定义是否为同义词，它们能否区分具有重要差异的相似概念，并在提示下甚至解释细微的差异。用于一般领域的大型语言模型的一个已知限制是，即使您基于错误的假设给它们输入，它们也总是会回答问题。我们将尝试了解我们是否为SE领域找到了类似的方面，例如，通过查看模型对询问它们可以使用哪些工具来执行自动手动测试或Haskell的最佳面向对象设计模式的提示做出的反应。通过我们的工作，我们不仅试图识别我们是否得到无稽之谈的响应，而且我们是否可以找到方法来推断生成的响应是无稽之谈，这在一般领域是可能的。此外，我们针对不同类型的模型研究上述方面：具有仅编码器的变压器架构的较小模型(例如，BERT)、仅编码器的较大模型(例如，Roberta)、具有允许更长上下文的变压器架构变化的模型(例如，Big Bird)、仅GPT样式的解码器的模型(例如，GPT-Neox)以及编码器-解码器模型(例如，T5)。我们将考虑特定于SE的预培训，以及基于一般领域数据培训的模型。由于一些一般的领域模型已经用包括SE数据的语料库进行了预训练，这也允许我们了解如果这只是非常大的数据集的一小部分，SE知识是否被充分捕获。

The transformer architecture has changed the field of Natural Language Processing (NLP) and paved the way for models such as BERT and GPT. These models have in common that they use transfer learning in the form of pre-training to learn a general representation of language, which can then be fine-tuned or prompted to perform various downstream tasks. While these models achieve remarkable results in a variety of NLP tasks, it is often unclear why they perform well on specific tasks and how well they work in different domains, such as Software Engineering (SE). Within our prior work, we looked at the impact of domain-specific pre-training on NLP tasks within the SE domain and we found that for polysemous words like "bug" (insect vs. defect) or "root" (plant vs. User), the domain-specific pre-training did help with the understanding of the meaning in the SE domain and that this also led to better performance in domain-specific downstream task. Within this project, we want to deepen our understanding of the capability of NLP models to capture concepts from the SE domain, with a focus on SE definitions and commonsense knowledge. We will use the analogy of NLP models as students to understand how they would perform in SE exams. For example, we will test if the NLP models contain accurate SE definitions and terminology: can the NLP models spot the correct definition of a term in a multiple-choice test, can they generate accurate definitions given a prompt, are they able to understand if definitions are synonyms, and can they differentiate between similar concepts with important differences and, given a prompt, even explain the small differences. A known limitation of large language models for the general domain is that they always answer, even if you give them inputs based on wrong assumptions. We will try to understand if we find similar aspects for the SE domain, e.g., by looking at how models react on prompts asking them which tools can be used to execute automated manual tests or what the best object-oriented design patterns for Haskell are. Through our work, we not only try to identify if we get nonsense responses, but also if we can find methods to infer that generated responses are nonsense, as is possible in the general domain. Additionally, we study the above aspects for different types of models: smaller models with an encoder-only transformer architecture (e.g., BERT), larger encoder-only models (e.g., RoBERTa), models with variations of the transformer architecture to allow for longer contexts (e.g., Big Bird), GPT-style decoder-only models (e.g., GPT-NeoX), and encoder-decoder models (e.g., T5). We will consider both SE-specific pre-training, as well as models trained on general domain data. Since some general domain models were already pre-trained with a corpus that included SE data this also allows us to understand if SE knowledge is sufficiently captured if this is only a smaller part of a very large data set.