CompCog: RI: Small: Human-like semantic grammar induction through knowledge distillation from pre-trained language models

CompCog：RI：Small：通过预训练语言模型的知识蒸馏进行类人语义语法归纳

• 批准号: 2313140
• 负责人: William Schuler
• 金额: $ 48.45万
• 依托单位: Ohio State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-09-01 至 2026-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2313140&HistoricalAwards=false
• 关键词: CompCog; RI; Small; Human; like

Human languages are thought to allow for an unbounded set of possible meanings to be expressed using bounded sets of rules, called grammars. These grammars assign meanings to words and compose meanings of words and phrases into larger phrases and clauses. Humans can communicate extremely precise descriptions of goals and world behaviors—and linguists know much about the logical structure of language involved in this kind of precise communication—but one of the central open questions in linguistics is how humans acquire these mechanisms. Computational models of how these grammars are learned from recorded or transcribed utterances can provide evidence that this learning can be accomplished by children without substantial innate biological biases, and these models can also provide automated tools for analysis and documentation of endangered languages, including many Indigenous American languages. Existing statistical and neural grammar learning methods can induce grammars from sentences in text corpora that predict about half of the phrases and clauses annotated by linguists; howevert this level of performance is nowhere near the accuracy of human language learners, and attempts to support this learning using image and video data have not substantially improved induction accuracy. The proposed work will instead extract statistics about logical predicates from large commercially available neural language models as a surrogate for human world knowledge so as to improve the accuracy of grammar induction.The proposed work will develop the first broad-coverage semantic grammar induction model that integrates world knowledge into the acquisition process by distilling it from large pre-trained neural language models. The world knowledge implicit in the large language models will be distilled into a matrix of predicate co-occurrence statistics using argument-specific prompts. The resulting predicate co-occurrence statistics will make no distinction between, for example, active and passive sentences, topicalized and non-topicalized sentences, or declarative and subject-auxiliary inverted sentences. This model will be used to evaluate claims about the statistical learnability of grammar. The proposed work will also continue work on developing resources for evaluating these structural models. The model and corpora collected as part of this project will be freely distributed on both university and external websites.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

人类语言被认为允许使用有限的规则集（称为语法）来表达无限的可能含义集。这些语法为单词赋予意义，并将单词和短语的意义组成更大的短语和分句。人类可以对目标和世界行为进行极其精确的描述——语言学家对这种精确交流所涉及的语言逻辑结构了解甚多——但语言学中一个悬而未决的核心问题是人类是如何获得这些机制的。这些语法是如何从记录或转录的话语中学习的计算模型可以提供证据，证明这种学习可以由儿童完成，而不存在重大的先天生物偏见，这些模型还可以为濒危语言（包括许多美洲土著语言）的分析和记录提供自动化工具。现有的统计和神经语法学习方法可以从文本语料库中的句子中归纳出语法，预测语言学家注释的大约一半的短语和分句；然而，这种表现水平与人类语言学习者的准确性相差甚远，并且尝试使用图像和视频数据来支持这种学习并没有实质性地提高归纳准确性。提议的工作将从大型商业可用的神经语言模型中提取有关逻辑谓词的统计数据，作为人类世界知识的替代品，从而提高语法归纳的准确性。提出的工作将开发第一个广泛覆盖的语义语法归纳模型，通过从大型预训练的神经语言模型中提取世界知识，将其集成到获取过程中。大型语言模型中隐含的世界知识将使用特定于参数的提示提取到谓词共现统计的矩阵中。结果谓词共现统计不会区分主动句和被动句、主题化句和非主题化句，或者陈述句和主辅助倒装句。该模型将用于评估关于语法统计可学习性的主张。拟议的工作还将继续为评估这些结构模型开发资源。作为这个项目的一部分，收集的模型和语料库将免费发布在大学和外部网站上。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。