SHF: Small: Open-domain, Data-driven Code Synthesis from Natural Language

SHF：小型：开放域、数据驱动的自然语言代码合成

• 批准号: 1815287
• 负责人: Graham Neubig
• 金额: $ 49.97万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-10-01 至 2022-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1815287&HistoricalAwards=false
• 关键词: SHF; Small; Open; domain; Data

One of the major hurdles in programming is turning ideas into code; all programmers, even experts, frequently reach points in a program where they know what they want to do but cannot easily turn it into a concrete implementation. In this case, it is common to turn to the web, e.g. enter a natural language query, search, browse results, copy-and-paste appropriate code, and modify it to the desired shape. However, this process is still time-consuming. This research aims to automate and enhance this process, by creating new data-driven methods for code synthesis from natural language, which allow developers to go directly from natural language description to code. Specifically, this project's goal is to bring code synthesis to the open domain, moving from highly engineered methods that work on only a single programming language or task, to methods that have the flexibility and scalability to answer most of the questions asked by programmers, in many different programming languages. The intellectual merit of this cross-disciplinary project lies in its potential to contribute to software engineering through the examination of developer's interaction with natural language productivity tools, and its potential to contribute to natural language processing through new models to understand procedural texts. This project will have broader impact through the development of tools and data linking together programs and natural language, potential to improve STEM education by lowering the barriers to programming, and training of graduate and undergraduate research assistants who will be able to straddle and act as bridges between the fields of natural language processing and software engineering.There are three technical pillars to the work. First, it will focus on methods to mine data consisting of natural language and corresponding code at scale, necessary for training. The mining will be performed over existing online data sources, such as community question answering sites (Stack Overflow) and open-source software repositories (GitHub), using machine learning models that consider both content matches and available meta-data, and crowd-sourcing-based verification of the extracted data. Second, the project will develop code synthesis methods that have the flexibility to handle the wide variety of expressions expected across a variety of software projects and developer needs. This will be done by developing models using neural networks, which have recently shown impressive ability to interpret a wide variety of expressions in other natural language processing tasks. We will expand these models to condition on project context, which will ensure handling of the various constraints necessary to create well-formed programs and allow for adaptation to project-specific conventions and needs. Third, the project will develop methods for learning and improving the models from developer behavior, by feeding back corrections to the generated code into the system and learning from the differences between the pre- and post-correction code. These methods will all be integrated into developer support tools that can be used in a development environment, or through an online API. The utility of these methods will be examined in both controlled and in-the-wild studies. Controlled studies will examine the subjective accuracy of the mined data and generated code, as well as the effect of the tools on the efficiency and ease of development, for programmers from novice to expert level. This project will also create and release tools for general consumption, solicit feedback from a wide variety of developers, and examine how developers use the proposed tools.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

编程中的一个主要障碍是将想法转化为代码;所有程序员，甚至专家，经常在程序中到达他们知道他们想要做什么的点，但不能轻易地将其转化为具体的实现。在这种情况下，通常转向web，例如输入自然语言查询、搜索、浏览结果、复制并粘贴适当的代码，并将其修改为所需的形状。然而，这一过程仍然很耗时。这项研究旨在通过创建新的数据驱动方法来自动化和增强这一过程，这些方法用于从自然语言合成代码，允许开发人员直接从自然语言描述到代码。具体来说，该项目的目标是将代码合成引入开放领域，从只适用于单一编程语言或任务的高度工程化方法，转变为具有灵活性和可扩展性的方法，以回答程序员在许多不同编程语言中提出的大多数问题。这个跨学科项目的智力价值在于它有可能通过检查开发人员与自然语言生产力工具的交互来促进软件工程，以及它有可能通过新模型来理解过程文本来促进自然语言处理。该项目将通过开发将程序和自然语言联系起来的工具和数据，通过降低编程障碍来改善STEM教育的潜力，以及培养能够跨越自然语言处理和软件工程领域并充当桥梁的研究生和本科生研究助理，产生更广泛的影响。首先，它将专注于挖掘由自然语言和相应代码组成的数据的方法，这是训练所必需的。挖掘将在现有的在线数据源上进行，如社区问答网站（Stack Overflow）和开源软件存储库（GitHub），使用机器学习模型，考虑内容匹配和可用的元数据，并基于众包对提取的数据进行验证。其次，该项目将开发代码合成方法，这些方法具有处理各种软件项目和开发人员需求中预期的各种表达的灵活性。这将通过使用神经网络开发模型来实现，神经网络最近在其他自然语言处理任务中表现出令人印象深刻的解释各种表达的能力。我们将扩展这些模型，以适应项目环境，这将确保处理创建格式良好的程序所需的各种约束，并允许适应特定于项目的约定和需求。第三，该项目将开发从开发人员行为中学习和改进模型的方法，方法是将对生成代码的更正反馈到系统中，并从更正前和更正后代码之间的差异中学习。这些方法都将集成到开发人员支持工具中，这些工具可以在开发环境中使用，也可以通过在线API使用。这些方法的效用将在对照和野生研究中进行检查。受控研究将检查挖掘数据和生成代码的主观准确性，以及工具对开发效率和易用性的影响，适用于从新手到专家级别的程序员。该项目还将创建和发布用于一般消费的工具，征求各种开发人员的反馈，并检查开发人员如何使用拟议的工具。该奖项反映了NSF的法定使命，并已被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估。

项目成果

Incorporating External Knowledge through Pre-training for Natural Language to Code Generation

• DOI: 10.18653/v1/2020.acl-main.538
• 发表时间: 2020-04
• 期刊: ArXiv
• 作者: Frank F. Xu;Zhengbao Jiang;Pengcheng Yin;Bogdan Vasilescu;Graham Neubig

How Can We Know What Language Models Know?

• DOI: 10.1162/tacl_a_00324
• 发表时间: 2020-01-01
• 期刊: TRANSACTIONS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS
• 影响因子: 10.9
• 作者: Jiang, Zhengbao;Xu, Frank F.;Neubig, Graham
• 通讯作者: Neubig, Graham

Retrieval-Based Neural Code Generation

• DOI: 10.18653/v1/d18-1111
• 发表时间: 2018-08
• 期刊: ArXiv
• 作者: Shirley Anugrah Hayati;R. Olivier;Pravalika Avvaru;Pengcheng Yin;A. Tomasic;Graham Neubig

Learning Structural Edits via Incremental Tree Transformations

• 发表时间: 2021-01
• 期刊: ArXiv
• 作者: Ziyu Yao;Frank F. Xu;Pengcheng Yin;Huan Sun;Graham Neubig

TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generation

• DOI: 10.18653/v1/d18-2002
• 发表时间: 2018-10
• 作者: Pengcheng Yin;Graham Neubig