CompCog: RI: Small: Human-like semantic grammar induction through knowledge distillation from pre-trained language models

人类语言被认为允许使用被称为语法的有界规则集来表达一组无限的可能意义。这些语法为单词赋予意义，并将单词和短语的意义组合成更大的短语和从句。人类可以非常精确地描述目标和世界行为--语言学家对这种精确交流所涉及的语言逻辑结构了如指掌--但语言学中的一个中心悬而未决的问题是，人类是如何获得这些机制的。这些语法是如何从记录或转录的话语中学习的计算模型可以提供证据，表明这种学习可以由儿童完成，而不存在实质性的先天生物偏见，这些模型还可以为分析和记录濒危语言提供自动化工具，包括许多土著美国语言。现有的统计和神经语法学习方法可以从文本语料库中的句子中归纳出语法，这些句子预测了语言学家标注的大约一半的短语和从句；然而，这种水平的表现与人类语言学习者的准确性相去甚远，使用图像和视频数据来支持这种学习的尝试并没有显著提高归纳的准确性。该工作将从大型商用神经语言模型中提取逻辑谓词的统计数据，作为人类世界知识的替代，从而提高语法归纳的准确性。该工作将开发第一个覆盖范围广泛的语义语法归纳模型，通过从大型预训练神经语言模型中提取世界知识来将世界知识整合到习得过程中。大型语言模型中隐含的世界知识将使用特定于论元的提示提炼成谓词共现统计矩阵。例如，所得到的谓词共现统计将不区分主动句和被动句、主题化和非主题化句子、陈述句和主助语倒排句。这个模型将被用来评估关于语法统计可学习性的主张。拟议的工作还将继续开发评价这些结构模型的资源。作为该项目的一部分，收集的模型和语料库将在大学和外部网站上免费分发。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Human languages are thought to allow for an unbounded set of possible meanings to be expressed using bounded sets of rules, called grammars. These grammars assign meanings to words and compose meanings of words and phrases into larger phrases and clauses. Humans can communicate extremely precise descriptions of goals and world behaviors—and linguists know much about the logical structure of language involved in this kind of precise communication—but one of the central open questions in linguistics is how humans acquire these mechanisms. Computational models of how these grammars are learned from recorded or transcribed utterances can provide evidence that this learning can be accomplished by children without substantial innate biological biases, and these models can also provide automated tools for analysis and documentation of endangered languages, including many Indigenous American languages. Existing statistical and neural grammar learning methods can induce grammars from sentences in text corpora that predict about half of the phrases and clauses annotated by linguists; howevert this level of performance is nowhere near the accuracy of human language learners, and attempts to support this learning using image and video data have not substantially improved induction accuracy. The proposed work will instead extract statistics about logical predicates from large commercially available neural language models as a surrogate for human world knowledge so as to improve the accuracy of grammar induction.The proposed work will develop the first broad-coverage semantic grammar induction model that integrates world knowledge into the acquisition process by distilling it from large pre-trained neural language models. The world knowledge implicit in the large language models will be distilled into a matrix of predicate co-occurrence statistics using argument-specific prompts. The resulting predicate co-occurrence statistics will make no distinction between, for example, active and passive sentences, topicalized and non-topicalized sentences, or declarative and subject-auxiliary inverted sentences. This model will be used to evaluate claims about the statistical learnability of grammar. The proposed work will also continue work on developing resources for evaluating these structural models. The model and corpora collected as part of this project will be freely distributed on both university and external websites.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.