An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation
An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation
复制标题
DOI：
10.1109/tse.2023.3334955
复制
发表时间：
2023-02
影响因子：
7.4
通讯作者：
Max Schäfer;Sarah Nadi;A. Eghbali;F. Tip
中科院分区：
文献类型：
--
作者：
Max Schäfer;Sarah Nadi;A. Eghbali;F. Tip
单元测试在确保软件的正确性方面起着关键作用。然而，手动创建单元测试是一项费力的任务，这激发了对自动化的需求。大型语言模型（llm）最近被应用到软件开发的各个方面，包括它们被建议用于自动生成单元测试，但同时需要额外的培训或对现有测试示例进行少量的学习。本文对llm在不需要额外的培训或手工工作的情况下自动生成单元测试的有效性进行了大规模的实证评估。具体地说，我们考虑一种方法，在这种方法中，LLM被提供提示，包括在测试中功能的签名和实现，以及从文档中提取的使用示例。此外，如果生成的测试失败，我们的方法将尝试生成一个新的测试，通过使用失败的测试和错误消息重新提示模型来修复问题。我们在TestPilot中实现了我们的方法，这是一个自适应的基于llm的JavaScript测试生成工具，可以自动为给定项目API中的方法生成单元测试。我们使用OpenAI的gpt3.5-turbo LLM在25个npm包上评估了TestPilot，总共有1,684个API函数。生成的测试的语句覆盖率中值为70.2% and branch coverage of 52.8%. In contrast, the state-of-the feedback-directed JavaScript test generation technique, Nessie, achieves only 51.3% statement coverage and 25.6% branch coverage. Furthermore, experiments with excluding parts of the information included in the prompts show that all components contribute towards the generation of effective test suites. We also find that 92.8% of TestPilot's generated tests have $\leq$≤ 50% similarity with existing tests (as measured by normalized edit distance), with none of them being exact copies. Finally, we run TestPilot with two additional LLMs, OpenAI's older code-cushman-002 LLM and StarCoder, an LLM for which the training process is publicly documented. Overall, we observed similar results with the former (68.2% median statement coverage), and somewhat worse results with the latter (54.0% median statement coverage), suggesting that the effectiveness of the approach is influenced by the size and training set of the LLM, but does not fundamentally depend on the specific model.
Unit tests play a key role in ensuring the correctness of software. However, manually creating unit tests is a laborious task, motivating the need for automation. Large Language Models (LLMs) have recently been applied to various aspects of software development, including their suggested use for automated generation of unit tests, but while requiring additional training or few-shot learning on examples of existing tests. This paper presents a large-scale empirical evaluation on the effectiveness of LLMs for automated unit test generation without requiring additional training or manual effort. Concretely, we consider an approach where the LLM is provided with prompts that include the signature and implementation of a function under test, along with usage examples extracted from documentation. Furthermore, if a generated test fails, our approach attempts to generate a new test that fixes the problem by re-prompting the model with the failing test and error message. We implement our approach in TestPilot, an adaptive LLM-based test generation tool for JavaScript that automatically generates unit tests for the methods in a given project's API. We evaluate TestPilot using OpenAI's gpt3.5-turbo LLM on 25 npm packages with a total of 1,684 API functions. The generated tests achieve a median statement coverage of 70.2% and branch coverage of 52.8%. In contrast, the state-of-the feedback-directed JavaScript test generation technique, Nessie, achieves only 51.3% statement coverage and 25.6% branch coverage. Furthermore, experiments with excluding parts of the information included in the prompts show that all components contribute towards the generation of effective test suites. We also find that 92.8% of TestPilot's generated tests have $\leq$≤ 50% similarity with existing tests (as measured by normalized edit distance), with none of them being exact copies. Finally, we run TestPilot with two additional LLMs, OpenAI's older code-cushman-002 LLM and StarCoder, an LLM for which the training process is publicly documented. Overall, we observed similar results with the former (68.2% median statement coverage), and somewhat worse results with the latter (54.0% median statement coverage), suggesting that the effectiveness of the approach is influenced by the size and training set of the LLM, but does not fundamentally depend on the specific model.