Collaborative Research: SHF: Medium: Natural Language Models with Execution Data for Software Testing

自然语言处理（NLP）模型已经被证明对各种软件工程任务非常有用，包括代码完成、注释生成和更新、代码审查生成和克隆检测。尽管软件测试在工业中很重要，但使用这些人工智能（AI）模型来开发和维护测试代码的工作很少，而测试代码是现实世界中软件测试的关键部分。测试代码与常规代码有多种不同之处：(1)测试代码以特定的方式结构化，包括设置测试环境和比较预期结果的步骤；(2)测试代码具有更丰富的上下文，例如它正在测试的具体方法和代码（被测代码）；(3)测试代码使用与被测代码不同的代码元素，即具有不同的控制结构；(4)测试代码有特定的输入值和预期结果；与常规代码不同，测试代码可以很容易地执行。这个项目的目标是通过简化测试开发和维护的NLP模型（NLP4Test）来提高软件工程师的生产力。具体来说，任务包括测试生成和完成、测试更新（当底层代码更改时），以及跨不同编程语言自动迁移测试。该项目探索测试通用代码库和新兴机器学习（ML）应用程序。这个项目的目标是一个新的领域——NLP4Test，这个领域需要创新的NLP模型。这个项目的结果将包括新技术、这些技术的实现，以及对开源项目的广泛评估。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Natural Language Processing (NLP) models have proven useful for various software engineering tasks, including code completion, comment generation and update, code review generation, and clone detection. Despite the importance of software testing in industry, there has been little work on using these Artificial Intelligence (AI) models for developing and maintaining test code, which is a key part of software testing in the real world. Test code differs in multiple ways from regular code: (1) Test code is structured in a specific way, with steps for setting up a test environment and comparing expected results; (2) Test code has richer context, such as the specific methods and code it is testing (code under test); (3) Test code uses different code elements than the code under test, i.e., it has a different control structure; (4) Test code has specific input values and expected results; (5) Unlike regular code, test code can be readily executed.The goal of this project is to increase the productivity of software engineers via NLP models that simplify the development and maintenance of tests (NLP4Test). Specifically, tasks include test generation and completion, test update (when the underlying code changes), and automatically migrating tests across different programming languages. This project explores testing both general codebases and emerging machine learning (ML) applications. The project targets a novel domain -- NLP4Test, and this domain requires innovative NLP models. The outcome of this project will include novel techniques, implementations of these techniques, and extensive evaluations on open-source projects.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.