CRI: CI-P: Collaborative: Towards a Program Analysis Collaboratory

基准程序在评价程序分析研究进展中起着重要作用。因此，评估的可信度取决于基准项目的数量和质量。他们的人数越多，越接近真实世界的项目，评估的置信度就越高。目前，程序分析领域的研究往往依赖于一些过时或有缺陷的基准程序进行评估。获得足够的基准程序的困难来自于程序分析器对基准程序的要求。通常，研究人员手动获取、检查和转换潜在的基准程序，以创建满足这些要求的基准程序。这种不可伸缩的方法往往无法产生足够的基准程序。这项研究的目标是使这一过程自动化，并通过在线的公共基础设施使其可用。该基础设施便于获取、选择和转换开源软件项目，以用作基准程序。此外，该基础设施使研究人员能够轻松地为程序分析管理基准程序，并共享根据这些基准运行分析器的结果。该项目的目标是从程序分析社区征求关于自动生成基准程序的要求和反馈，并构建实现该过程的程序分析协作实验室(PACLab)研究基础设施的原型。PACLab使用研究人员关于充分基准程序生成的规范，在开源存储库中定位和获取潜在的基准程序，根据研究人员的指定执行必要的程序转换，并输出适当的基准程序。此外，PACLab使研究人员能够使用容器技术共享他们的程序分析器、转换后的基准程序以及在这些基准程序上运行他们的分析器的结果。该项目的智力价值在于研究为目标程序分析器指定和自动化程序转换的技术。该项目的更广泛影响来自于它有可能通过提高对方案分析进展的评价的信心来扩大其适应；降低新的方案分析研究人员的准入门槛；以及加快方案分析研究的步伐。PACLab的教育好处包括让学生能够轻松定位真实世界的项目并评估他们对项目分析器的实施情况。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Benchmark programs play an important role in evaluating the advances in program analysis research. Thus, the confidence of the evaluation depends on the quantity and the quality of the benchmark programs. The larger their number and the closer they resemble real-world programs, the higher confidence in the evaluation. Currently, research in the field of program analysis often relies on a few outdated or deficient benchmark programs for evaluation purposes. The difficulty in obtaining sufficient benchmark programs comes from the requirements imposed on the benchmark programs by program analyzers. Usually researchers manually obtain, inspect, and transform potential benchmark programs to create benchmark programs that meet those requirements. Such an unscalable approach often fails to produce adequate benchmark programs. The goal of this research is to automate this process and make it available through an online, public infrastructure. The infrastructure facilitates obtaining, selecting, and transforming open-source software projects for use as benchmark programs. In addition, the infrastructure gives researchers the ability to easily curate benchmark programs for program analysis and share the results of running their analyzers on those benchmarks. The objectives of this project are to solicit requirements and feedback from the program analysis community on automating benchmark program generation, and to prototype a Program Analysis Collaboratory (PAClab) research infrastructure that implements that process. Using researchers' specifications for adequate benchmark program generation, PAClab locates and obtains potential benchmark programs in open-source repositories; performs necessary program transformations as specified by researchers; and outputs the adequate benchmark programs. In addition, PAClab enables researchers to share their program analyzers using container technology, the transformed benchmark programs, and the results of running their analyzers on those benchmark programs. The intellectual merit of the project lies in investigating techniques for specifying and automating program transformations for targeted program analyzers. The broader impacts of the project stem from its potential to broaden adaptation of program analysis advances through the increased confidence of their evaluations; to lower the entry barrier for new program analysis researchers; and to accelerate the pace in program analysis research. PAClab's educational benefits include the ability for students to easily locate real-world programs and evaluate their implementations of program analyzers.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.