CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
复制标题
DOI：
--
复制
发表时间：
2019-09
期刊：
影响因子：
--
通讯作者：
Hamel Husain;Hongqiu Wu;Tiferet Gazit;Miltiadis Allamanis;Marc Brockschmidt
中科院分区：
文献类型：
--
作者：
Hamel Husain;Hongqiu Wu;Tiferet Gazit;Miltiadis Allamanis;Marc Brockschmidt
语义代码搜索是在给定的自然语言查询中检索相关代码的任务。虽然与其他信息检索任务相关，但它需要弥合代码中使用的语言（通常是缩写的和高度技术性的）与更适合描述模糊概念和想法的自然语言之间的差距。为了评估代码搜索的进展，我们发布了CodeSearchNet语料库，并提出了CodeSearchNet挑战，该挑战由99个自然语言查询和大约4k专家相关注释组成，这些注释来自CodeSearchNet语料库的可能结果。该语料库包含大约600万个函数，这些函数来自六种编程语言（Go、Java、JavaScript、PHP、Python和Ruby）的开源代码。CodeSearchNet语料库还包含自动生成的类似查询的自然语言，用于200万个函数，这些函数是通过机械抓取和预处理相关函数文档获得的。在本文中，我们描述了用于获取语料库和专家标签的方法，以及该任务的一些简单基线解决方案。我们希望CodeSearchNet挑战赛鼓励研究人员和实践者进一步研究这一有趣的任务，并将举办比赛和排行榜来跟踪挑战的进展。我们也热衷于在未来将CodeSearchNet挑战扩展到更多的查询和编程语言。
Semantic code search is the task of retrieving relevant code given a natural language query. While related to other information retrieval tasks, it requires bridging the gap between the language used in code (often abbreviated and highly technical) and natural language more suitable to describe vague concepts and ideas. To enable evaluation of progress on code search, we are releasing the CodeSearchNet Corpus and are presenting the CodeSearchNet Challenge, which consists of 99 natural language queries with about 4k expert relevance annotations of likely results from CodeSearchNet Corpus. The corpus contains about 6 million functions from open-source code spanning six programming languages (Go, Java, JavaScript, PHP, Python, and Ruby). The CodeSearchNet Corpus also contains automatically generated query-like natural language for 2 million functions, obtained from mechanically scraping and preprocessing associated function documentation. In this article, we describe the methodology used to obtain the corpus and expert labels, as well as a number of simple baseline solutions for the task. We hope that CodeSearchNet Challenge encourages researchers and practitioners to study this interesting task further and will host a competition and leaderboard to track the progress on the challenge. We are also keen on extending CodeSearchNet Challenge to more queries and programming languages in the future.