NSF-NSERC: SaTC: CORE: Small: Managing Risks of AI-generated Code in the Software Supply Chain

现代软件是通过将预先存在的软件包组合成软件产品来创建的。这种方法是由于开源范例的日益流行而实现的，在这种范例中，软件包的源代码是在允许重用的许可证下提供的。这种方法加快了软件开发速度，带来了显著的经济效益，但也带来了无意中将易受攻击的代码导入关键软件工具的风险。在开源开发中，越来越多地使用人工智能(AI)工具来生成代码，这进一步加剧了风险。这些工具必须对大量数据进行培训，而这些数据并不总是经过严格审查，因此它们可能会学习生成易受攻击的代码。更糟糕的是，恶意方可能会主动向其训练集中注入恶意代码。不幸的是，人们对所有这些问题仍然知之甚少。该项目旨在衡量和减轻软件供应链中人工智能生成的代码所产生的风险。它将调查人工智能工具的使用有多普遍，并表征它们带来的安全风险。通过这样做，它将解决紧迫的经济和社会需求：人工智能承诺为软件开发带来显著的好处，但只有在其风险得到缓解的情况下，这些才能实现。研究成果将通过研讨会和黑客松传播，结果将成为课程和课程的一部分。这项工作将通过生产来源工具来提高软件供应链的安全性，从而使开放源码社区受益。该项目是与来自加拿大的研究人员合作的，具有互补的专业知识，为该项目提供额外的资源。从技术上讲，正在研究的AI工具由各种用于代码生成的大型语言模型(LLM)组成。感兴趣的威胁模型是开发人员将易受攻击的LLM生成的代码插入到安全关键型程序中的模型，无论是由于低质量的代码生成还是使用有毒/后备的LLM。该项目包括三个推力，每个推力都解决了与威胁模型相关的一个研究问题：(I)如何以及在多大程度上可以将LLM代码与人类编写的代码区分开来；(Ii)LLM代码已经在多大程度上存在于供应链中，其安全影响是什么；以及(Iii)在现实条件下，针对LLM代码生成的中毒攻击可以在多大程度上成功。在推力(I)中，该项目将现有的代码样式技术(直到现在用于区分人类程序员)扩展到区分人类和LLM生成的代码的新问题。在《推力(II)》一书中，调查人员对开源软件进行了度量研究，对LLM生成的代码在供应链中的存在和影响产生了经验上的理解。最后，推力(III)着眼于代码后门的实际可行性，以及基于声誉的自动化审查作为辩护的有效性。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Modern software is created by combining pre-existing software packages into a software product. This approach is enabled by the growing popularity of the Open-Source paradigm, where the source code of software packages is made available under licenses that allow reuse. This approach speeds up software development with significant economic benefits, but also creates the risk of inadvertently importing vulnerable code into critical software tools. The risk is further compounded by the increasing use of Artificial Intelligence (AI) tools for code generation in Open-Source development. These tools must be trained on enormous amounts of data, which is not always rigorously reviewed, and thus they may learn to generate vulnerable code. To make matters worse, malicious parties may actively inject malicious code in their training set. Unfortunately, all these issues are still poorly understood. This project aims at measuring and mitigating the risks emerging from AI-generated code in the software supply chain. It will investigate how prevalent the use of AI tools is, and characterize the security risks they entail. In doing so, it will address pressing economic and societal needs: AI promises to bring significant benefits to software development, but those can only be achieved if its risks are mitigated. The research outcomes will be disseminated through workshops and hackathons, and the results will become part of curriculum and courses. The work will benefit the open-source community by producing provenance tools to improve software supply chain security. The project is a collaboration with researchers from Canada with complementary expertise that provides additional resources to the project. Technically, the AI tools being investigated consist of various Large Language Models (LLM) for code generation. The threat model of interest is one where a developer inserts vulnerable LLM-generated code into a security-critical program, be it due to low-quality code generation or using a poisoned/backdoored LLM. This project consists of three thrusts, each addressing a research question relevant to the threat model: (i) how, and to what extent, LLM code can be distinguished from code written by humans; (ii) to what extent LLM code is already present in the supply chain, and what are its security implications; and (iii) to what extent poisoning attacks against LLM code generation can succeed in realistic conditions. In thrust (i), this project extends existing code stylometry techniques, until now used to distinguish human programmers, to the novel problem of distinguishing human- and LLM-generated code. In thrust (ii), the investigators conduct measurement studies of Open-Source software, generating empirical understanding of the presence and implications of LLM-generated code in the supply chain. Finally, thrust (iii) looks at the practical feasibility of code backdoors, and the effectiveness of automated reputation-based vetting as a defense.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.