Collaborative Research: Inductive Biases for the Acquisition of Syntactic Transformations in Neural Networks

合作研究：神经网络中句法转换习得的归纳偏差

• 批准号: 1919321
• 负责人: Robert Frank
• 金额: $ 32.75万
• 依托单位: Yale University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-08-01 至 2024-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1919321&HistoricalAwards=false
• 关键词: Collaborative; Research; Inductive; Biases; Acquisition

Children are prodigious language learners; they quickly master the words and rules that govern the languages spoken by the communities in which they are raised. In contrast, modern artificial intelligence systems like Siri and Alexa need to be trained extensively on massive data sets; even then the linguistic abilities of such systems lag far behind those of a child. Scientific research on language acquisition and structure has taught us that children come to the task of language learning with preconceptions of what their language will look like. They use such preconceptions, referred to as "biases", to guide their learning, favoring language structures that are compatible with those biases. By structuring computers systems to incorporate these biases, we could construct computer interfaces that would be more effective not only for languages such as English and Spanish, but also for the many languages where training data is scarce, spoken in smaller communities within the United States and internationally. Moreover, understanding more about how the biases necessary for language learning can be instantiated in a computer model will help to resolve a long-standing debate about the nature of these human biases: are they specific to language or are they the result of more general properties of human cognition?The current project explores the learning of regularities in natural language syntax by computer systems. The focus will be specifically on systems based on neural networks, which have been behind the recent revolutionary advances in language technologies. The project will study a wide range of neural network architectures, some with explicitly represented linguistic biases and some not, and compare them with respect to their abilities to learn carefully defined linguistic patterns. In contrast to past work that has evaluated linguistic knowledge of neural networks indirectly through a language modeling (word prediction) task, this project instead explores tasks that are formulated as transformations, which map one linguistic form to another (e.g., question formation, verbal inflection, negation, passivization, mapping to logical form). Not only are such mappings at the basis of a widely applied class of neural network architectures, so-called sequence to sequence networks, but they are also a common way of characterizing syntactic processes in linguistics. As a result, the use of such mappings allows a more direct assessment of the networks' linguistic abilities. Part of the project will involve the collaborative development of training and testing datasets for the mappings, with involvement by an interdisciplinary team of linguists and computer scientists, and these will be made available as a resource for the entire research community. These datasets will then be used as the basis for the detailed analysis of neural network representations of linguistic structure. Furthermore, explicit comparisons will be carried out between neural network and human performance on the mapping tasks under study.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

孩子们是了不起的语言学习者；他们很快就掌握了支配他们成长所在社区所说语言的词汇和规则。相比之下，像Siri和Alexa这样的现代人工智能系统需要在海量数据集上接受广泛的培训；即使这样，这些系统的语言能力也远远落后于儿童。关于语言习得和结构的科学研究告诉我们，孩子们开始学习语言的时候，会对他们的语言会是什么样子有先入为主的概念。他们使用这种被称为“偏见”的先入为主的概念来指导他们的学习，偏爱与这些偏见相适应的语言结构。通过构建包含这些偏见的计算机系统，我们可以构建不仅对英语和西班牙语等语言更有效的计算机接口，而且对美国国内和国际上较小社区使用的训练数据稀缺的多种语言也是如此。此外，更多地了解如何在计算机模型中实例化语言学习所需的偏见，将有助于解决关于这些人类偏见的性质的长期争论：这些偏见是语言特有的，还是人类认知更普遍属性的结果？当前的项目探索了计算机系统学习自然语言句法中的规则。重点将特别放在基于神经网络的系统上，这些系统一直是最近语言技术革命性进步的幕后推手。该项目将研究广泛的神经网络结构，其中一些具有明确表示的语言偏见，另一些没有，并就它们学习仔细定义的语言模式的能力进行比较。与过去通过语言建模(单词预测)任务间接评估神经网络的语言知识的工作不同，该项目探索的是被表述为转换的任务，这些转换将一种语言形式映射到另一种语言形式(例如，问题形成、动词词形变化、否定、被动化、到逻辑形式的映射)。这种映射不仅是一类广泛应用的神经网络结构的基础，也是语言学中表征句法过程的一种常见方法。因此，使用这样的映射可以更直接地评估网络的语言能力。该项目的一部分将涉及在一个由语言学家和计算机科学家组成的跨学科小组的参与下，协作开发地图的培训和测试数据集，这些数据集将作为整个研究界的资源提供。这些数据集将被用作详细分析语言结构的神经网络表示的基础。此外，还将对神经网络和人类在所研究的映射任务中的表现进行明确的比较。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Subject-verb Agreement with Seq2Seq Transformers: Bigger Is Better, but Still Not Best

与 Seq2Seq Transformer 的主谓一致：越大越好，但仍不是最好

• 发表时间: 2023
• 期刊: Proceedings of the Society for Computation in Linguistics
• 作者: Wilson, Michael A.;Zhou, Zhenghao;and Frank, Robert
• 通讯作者: and Frank, Robert

Sequence-to-Sequence Networks Learn the Meaning of Reflexive Anaphora

• 发表时间: 2020-11
• 期刊: ArXiv
• 作者: R. Frank;Jackson Petty

Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexity

• DOI: 10.1162/tacl_a_00490
• 发表时间: 2022-04
• 期刊: Transactions of the Association for Computational Linguistics
• 影响因子: 10.9
• 作者: Sophie Hao;D. Angluin;R. Frank

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

• 发表时间: 2022-06
• 期刊: ArXiv
• 作者: Aarohi Srivastava;Abhinav Rastogi;Abhishek Rao;Abu Awal Md Shoeb;Abubakar Abid;Adam Fisch;Adam R. Brown-Ad

What affects Priming Strength? Simulating Structural Priming Effect with PIPS

什么影响底漆强度？

• 发表时间: 2023
• 期刊: Proceedings of the Society for Computation in Linguistics
• 作者: Zhou, Zhenghao;Frank, Robert
• 通讯作者: Frank, Robert