SHF: Small: Automated Unit Test Generation using Large Language Models

大型语言模型（LLM）是依靠机器学习技术来解决使用自然语言陈述的查询的系统。为了与LLM交互，用户提供待解决问题的文本描述，通常称为“提示”。响应于接收到提示，LLM生成表示问题的解决方案的一个或多个文本结果。LLM最近在翻译和聊天机器人等应用程序中非常受欢迎。此外，LLM开始被用于软件开发中的应用程序，例如程序员使用的集成开发环境中的自动代码完成。这个项目探讨了使用LLM自动测试软件。为此，通过向LLM提供包含被测软件的代码片段和与软件相关的工件（如文档和使用示例）的提示来生成测试。该项目将改进软件测试，从而提高软件质量。此外，该项目将减轻开发人员手动编写测试的负担，从而提高他们的生产力。基于大型语言模型（LLM）开发的自动测试生成技术将采用反馈导向的迭代方法。在每次迭代中，LLM都会得到一个提示，其中包含被测函数的签名，沿着使用示例，测试框架代码和从先前生成的测试中获得的结果。最终的完成由候选测试组成，这些测试被检查语法有效性，并被执行以检查它们是通过还是失败。对失败测试的执行行为的分析将指导精细提示的构造。该项目将针对动态类型和静态类型的编程语言。将通过将生成的测试套件与最先进的测试生成工具生成的测试套件进行比较来评估这项工作。所有开发的测试生成工具都将作为开放源码软件提供给其他人使用和借鉴，结果将通过出版物传播。通过采用所开发的技术，软件质量的提高将带来社会效益。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Large Language Models (LLMs) are systems that rely on machine learning techniques to solve queries that are stated using natural language. To interact with an LLM, a user provides a textual description, commonly referred to as a "prompt", of the problem to be solved. In response to receiving a prompt, an LLM generates one or more textual results that represent solutions to the problem. LLMs have recently become very popular for applications such as translation and chatbots. Furthermore, LLMs are starting to be adopted for applications in software development such as automatic code completion in Integrated Development Environments used by programmers. This project explores the use of LLMs for automatically testing software. To this end, tests are generated by providing LLMs with prompts containing code fragments of the software under test and artifacts associated with the software such as documentation and usage examples. The project will result in improved testing of software, thereby improving its quality. Moreover, the project will unburden developers from the chore of writing tests manually, thereby increasing their productivity. The developed automatic test generation techniques based on Large Language Models (LLMs) will feature a feedback-directed, iterative approach. In each iteration, an LLM is given a prompt containing the signature of a function under test, along with usage examples, test framework code, and results obtained from previously generated tests. The resulting completions consist of candidate tests, which are checked for syntactic validity and executed to check if they pass or fail. Analysis of the execution behavior of failing tests will direct the construction of refined prompts. The project will target both dynamically typed and statically typed programming languages. The work will be evaluated by comparing the generated test suites against those produced by state-of-the-art test generation tools. All developed test generation tools will be made available as open-source software for others to use and build on, and results will be disseminated via publications. Societal benefits will follow from improvements in software quality enabled by the adoption of the developed techniques.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.