CAREER: Insertion-Based Natural Language Generation

职业：基于插入的自然语言生成

• 批准号: 2339766
• 负责人: Nanyun Peng
• 金额: $ 58.55万
• 依托单位: University of California-Los Angeles
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-05-01 至 2029-04-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2339766&HistoricalAwards=false
• 关键词: CAREER; Insertion; Based; Natural; Language

AbstractLanguage models (LMs) have become the foundations of most natural language processing (NLP) applications nowadays. However, existing language models predominantly follow the auto-regressive paradigm, which trains models to predict the next word given the left-side context. Then, they generate sentences word-by-word, left-to-right. The simplicity of this paradigm is attractive but it has several limitations, including inefficiency in generation, lack of reliable control for human-machine collaboration, and more importantly, it significantly deviates from how humans interpret and compose sentences. The proposed research explores a fundamentally distinct paradigm of language modeling --- insertion-based models. Such models formulate the generation process as iteratively inserting words into an incomplete context, offering significantly more flexibility and controllability compared to the auto-regressive models. The insertion-based formulation better mimics human writing behaviors and thus provides a tool for computational linguistics and cognitive science to study the structure of languages. The controllability of the model will benefit communication researchers, content creators, and creative composers for applications such as creative content generation, tailored communication, and personalized user experiences. The research results will be integrated in teaching materials to disseminate generative artificial intelligence (AI) for K-12 and undergraduate. This project aims to advance our understandings of the benefits and capability of insertion-based LMs. The proposed research will explore different generation orders supported by the flexibility of the insertion-based formulation, with the goal to discover optimal insertion orders guided by linguistic theories. Additionally, the project will introduce a novel model architecture for a variation of insertion-based LMs that incorporates deletion operations, enabling the models to rectify previous generation errors. This enhancement brings added flexibility and controllability to insertion-based LMs. Furthermore, an ambitious goal of this research is to investigate the scaling law and scale up the pretraining of insertion-based LMs. The success of these explorations has the potential to lead to a family of large language models that exhibit enhanced flexibility, controllability, and inference efficiency surpassing the auto-regressive LMs, resulting in benefits for a wide range of natural language generation and general NLP tasks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

摘要语言模型（LM）已成为当今大多数自然语言处理（NLP）应用的基础。然而，现有的语言模型主要遵循自回归范式，该范式训练模型在给定左侧上下文的情况下预测下一个单词。然后，他们从左到右逐字逐句地生成句子。这种范式的简单性很有吸引力，但它有几个局限性，包括生成效率低下，缺乏对人机协作的可靠控制，更重要的是，它严重偏离了人类解释和撰写句子的方式。本研究探索了一种完全不同的语言建模范式-基于插入的模型。这种模型将生成过程表述为迭代地将单词插入到不完整的上下文中，与自回归模型相比，提供了更大的灵活性和可控性。这种基于插入的表达方式更好地模拟了人类的书写行为，为计算语言学和认知科学研究语言结构提供了一种工具。该模型的可控性将有利于通信研究人员，内容创作者和创意作曲家的应用，如创意内容生成，定制通信和个性化用户体验。研究成果将被整合到教材中，为K-12和本科生传播生成式人工智能（AI）。本项目旨在提高我们对基于插入的LM的优点和能力的理解。该研究将探索不同的生成顺序支持的灵活性的插入为基础的配方，以发现最佳的插入顺序的语言学理论指导的目标。此外，该项目将为基于插入的LM的变体引入一种新的模型架构，该架构包含删除操作，使模型能够纠正上一代错误。这种增强为基于插入的LM带来了额外的灵活性和可控性。此外，本研究的一个雄心勃勃的目标是研究缩放律并扩大基于插入的LM的预训练。这些探索的成功有可能导致一系列大型语言模型，这些模型表现出增强的灵活性，可控性和推理效率，超过自回归LM，该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的评估被认为值得支持。影响审查标准。