Collaborative Research: Inductive Biases for the Acquisition of Syntactic Transformations in Neural Networks

合作研究：神经网络中句法转换习得的归纳偏差

• 批准号: 2114505
• 负责人: Tal Linzen
• 金额: $ 38.11万
• 依托单位: New York University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2114505&HistoricalAwards=false
• 关键词: Collaborative; Research; Inductive; Biases; Acquisition

Children are prodigious language learners; they quickly master the words and rules that govern the languages spoken by the communities in which they are raised. In contrast, modern artificial intelligence systems like Siri and Alexa need to be trained extensively on massive data sets; even then the linguistic abilities of such systems lag far behind those of a child. Scientific research on language acquisition and structure has taught us that children come to the task of language learning with preconceptions of what their language will look like. They use such preconceptions, referred to as "biases", to guide their learning, favoring language structures that are compatible with those biases. By structuring computers systems to incorporate these biases, we could construct computer interfaces that would be more effective not only for languages such as English and Spanish, but also for the many languages where training data is scarce, spoken in smaller communities within the United States and internationally. Moreover, understanding more about how the biases necessary for language learning can be instantiated in a computer model will help to resolve a long-standing debate about the nature of these human biases: are they specific to language or are they the result of more general properties of human cognition?The current project explores the learning of regularities in natural language syntax by computer systems. The focus will be specifically on systems based on neural networks, which have been behind the recent revolutionary advances in language technologies. The project will study a wide range of neural network architectures, some with explicitly represented linguistic biases and some not, and compare them with respect to their abilities to learn carefully defined linguistic patterns. In contrast to past work that has evaluated linguistic knowledge of neural networks indirectly through a language modeling (word prediction) task, this project instead explores tasks that are formulated as transformations, which map one linguistic form to another (e.g., question formation, verbal inflection, negation, passivization, mapping to logical form). Not only are such mappings at the basis of a widely applied class of neural network architectures, so-called sequence to sequence networks, but they are also a common way of characterizing syntactic processes in linguistics. As a result, the use of such mappings allows a more direct assessment of the networks' linguistic abilities. Part of the project will involve the collaborative development of training and testing datasets for the mappings, with involvement by an interdisciplinary team of linguists and computer scientists, and these will be made available as a resource for the entire research community. These datasets will then be used as the basis for the detailed analysis of neural network representations of linguistic structure. Furthermore, explicit comparisons will be carried out between neural network and human performance on the mapping tasks under study.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

儿童是惊人的语言学习者;他们很快掌握了他们成长的社区所讲的语言的单词和规则。相比之下，像Siri和Alexa这样的现代人工智能系统需要在大量数据集上进行广泛的训练;即使这样，这些系统的语言能力也远远落后于儿童。关于语言习得和语言结构的科学研究告诉我们，孩子们在学习语言时，对他们的语言会是什么样子有先入为主的看法。他们使用这种被称为“偏见”的先入之见来指导他们的学习，偏爱与这些偏见相容的语言结构。通过构建计算机系统来整合这些偏见，我们可以构建更有效的计算机界面，不仅适用于英语和西班牙语等语言，而且适用于训练数据稀缺的许多语言，这些语言在美国和国际上的较小社区中使用。此外，更多地了解语言学习所需的偏见如何在计算机模型中实例化，将有助于解决关于这些人类偏见性质的长期争论：它们是语言特有的，还是人类认知更普遍属性的结果？目前的项目探讨了计算机系统在自然语言语法中的学习。重点将特别放在基于神经网络的系统上，这些系统是最近语言技术革命性进展的背后。该项目将研究广泛的神经网络架构，有些具有明确的语言偏见，有些则没有，并比较它们学习精心定义的语言模式的能力。与过去通过语言建模（单词预测）任务间接评估神经网络语言知识的工作相反，该项目探索了被公式化为转换的任务，将一种语言形式映射到另一种语言形式（例如，疑问句的形成，动词的变化，否定，被动化，映射到逻辑形式）。这种映射不仅是广泛应用的一类神经网络架构（所谓的序列到序列网络）的基础，而且它们也是语言学中表征句法过程的常见方式。因此，使用这种映射可以更直接地评估网络的语言能力。该项目的一部分将涉及在一个由语言学家和计算机科学家组成的跨学科小组的参与下，合作开发映射的培训和测试数据集，这些数据集将作为整个研究界的资源提供。 然后，这些数据集将被用作详细分析语言结构的神经网络表示的基础。此外，将对神经网络和人类在所研究的绘图任务上的表现进行明确的比较。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

COGS: A Compositional Generalization Challenge Based on Semantic Interpretation

• DOI: 10.18653/v1/2020.emnlp-main.731
• 发表时间: 2020-10
• 期刊: ArXiv
• 作者: Najoung Kim;Tal Linzen

How to Plant Trees in LMs: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases

如何在语言模型中种树：数据和架构对句法归纳偏差出现的影响

• 发表时间: 2023
• 期刊: Proceedings of the conference Association for Computational Linguistics Meeting
• 作者: Mueller, Aaron;Linzen, Tal
• 通讯作者: Linzen, Tal

Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models

为空白石板着色：预训练为序列到序列模型赋予分层归纳偏差

• 发表时间: 2022
• 期刊: Findings of the Association for Computational Linguistics: ACL 2022
• 作者: Mueller, Aaron;Frank, Robert;Linzen, Tal;Wang, Luheng;Schuster, Sebastian
• 通讯作者: Schuster, Sebastian

Structure Here, Bias There: Hierarchical Generalization by Jointly Learning Syntactic Transformations

这里的结构，那里的偏见：通过联合学习句法转换进行层次化概括

• DOI: 10.7275/j0es-xf97
• 发表时间: 2021
• 期刊: Proceedings of the Society for Computation in Linguistics
• 作者: Karl Mulligan, Robert Frank

Syntactic Structure from Deep Learning

• DOI: 10.1146/annurev-linguistics-032020-051035
• 发表时间: 2021-01-01
• 期刊: ANNUAL REVIEW OF LINGUISTICS, VOL 7
• 作者: Linzen, Tal;Baroni, Marco
• 通讯作者: Baroni, Marco