SantaCoder: don't reach for the stars!
SantaCoder: don't reach for the stars!
复制标题
DOI：
10.48550/arxiv.2301.03988
复制
发表时间：
2023-01
期刊：
影响因子：
--
通讯作者：
Loubna Ben Allal;Raymond Li;Denis Kocetkov;Chenghao Mou;Christopher Akiki;Carlos Muñoz Ferrandis
中科院分区：
文献类型：
--
作者：
Loubna Ben Allal;Raymond Li;Denis Kocetkov;Chenghao Mou;Christopher Akiki;Carlos Muñoz Ferrandis
BigCode项目是一个开放的科学合作项目，致力于负责任地开发代码的大型语言模型。这份技术报告描述了截至2022年12月的协作进展，概述了个人身份信息(PII)编校管道的当前状态，为降低模型架构的风险而进行的实验，以及研究更好的训练数据预处理方法的实验。我们在Stack的Java、JavaScript和Python子集上训练1.1B参数模型，并在MultiPL-E文本到代码基准测试中对它们进行评估。我们发现，更积极地过滤近重复项可以进一步提高性能，令人惊讶的是，从具有5+GitHub星的存储库中选择文件会显著降低性能。我们最好的模型在从左到右的生成和填充多语言代码生成部分方面都优于以前的开源多语言代码生成模型(InCoder-6.7B和CodeGen-Multi-2.7B)，尽管它是一个小得多的模型。所有型号均在https://hf.co/bigcode.的OpenRAIL许可下发布
The BigCode project is an open-scientific collaboration working on the responsible development of large language models for code. This tech report describes the progress of the collaboration until December 2022, outlining the current state of the Personally Identifiable Information (PII) redaction pipeline, the experiments conducted to de-risk the model architecture, and the experiments investigating better preprocessing methods for the training data. We train 1.1B parameter models on the Java, JavaScript, and Python subsets of The Stack and evaluate them on the MultiPL-E text-to-code benchmark. We find that more aggressive filtering of near-duplicates can further boost performance and, surprisingly, that selecting files from repositories with 5+ GitHub stars deteriorates performance significantly. Our best model outperforms previous open-source multilingual code generation models (InCoder-6.7B and CodeGen-Multi-2.7B) in both left-to-right generation and infilling on the Java, JavaScript, and Python portions of MultiPL-E, despite being a substantially smaller model. All models are released under an OpenRAIL license at https://hf.co/bigcode.