Collaborative Research: Inductive Biases for the Acquisition of Syntactic Transformations in Neural Networks

儿童是惊人的语言学习者；他们很快就掌握了他们成长的社区所使用的语言的词汇和规则。相比之下，像Siri和Alexa这样的现代人工智能系统需要在海量数据集上进行广泛的训练；即便如此，这些系统的语言能力也远远落后于儿童。关于语言习得和语言结构的科学研究告诉我们，孩子们在开始语言学习的时候，对他们的语言会是什么样子有先入为主的观念。他们使用这种被称为“偏见”的先入为主的观念来指导他们的学习，偏爱与这些偏见相容的语言结构。通过构建包含这些偏差的计算机系统，我们可以构建计算机界面，不仅对英语和西班牙语等语言更有效，而且对许多训练数据稀缺的语言也更有效，这些语言在美国国内和国际上较小的社区中使用。此外，更多地了解语言学习所需的偏见是如何在计算机模型中实例化的，将有助于解决关于这些人类偏见本质的长期争论：它们是特定于语言的，还是人类认知的更普遍属性的结果？目前的项目是探索计算机系统对自然语言语法规律的学习。重点将特别放在基于神经网络的系统上，神经网络是最近语言技术革命性进步的背后推手。该项目将研究范围广泛的神经网络架构，其中一些具有明确表示的语言偏差，而有些则没有，并将它们与学习仔细定义的语言模式的能力进行比较。与过去通过语言建模（单词预测）任务间接评估神经网络的语言知识的工作相反，该项目探索了被表述为转换的任务，将一种语言形式映射到另一种语言形式（例如，问题形成、言语变化、否定、钝化、映射到逻辑形式）。这种映射不仅是广泛应用的一类神经网络架构（所谓的序列到序列网络）的基础，而且也是语言学中表征句法过程的常用方法。因此，使用这种映射可以更直接地评估网络的语言能力。该项目的一部分内容将包括协作开发用于映射的训练和测试数据集，由语言学家和计算机科学家组成的跨学科团队将参与其中，这些数据集将作为整个研究界的资源提供。这些数据集将被用作语言结构的神经网络表示的详细分析的基础。此外，将在神经网络和人类在研究中的映射任务上的表现进行明确的比较。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Children are prodigious language learners; they quickly master the words and rules that govern the languages spoken by the communities in which they are raised. In contrast, modern artificial intelligence systems like Siri and Alexa need to be trained extensively on massive data sets; even then the linguistic abilities of such systems lag far behind those of a child. Scientific research on language acquisition and structure has taught us that children come to the task of language learning with preconceptions of what their language will look like. They use such preconceptions, referred to as "biases", to guide their learning, favoring language structures that are compatible with those biases. By structuring computers systems to incorporate these biases, we could construct computer interfaces that would be more effective not only for languages such as English and Spanish, but also for the many languages where training data is scarce, spoken in smaller communities within the United States and internationally. Moreover, understanding more about how the biases necessary for language learning can be instantiated in a computer model will help to resolve a long-standing debate about the nature of these human biases: are they specific to language or are they the result of more general properties of human cognition?The current project explores the learning of regularities in natural language syntax by computer systems. The focus will be specifically on systems based on neural networks, which have been behind the recent revolutionary advances in language technologies. The project will study a wide range of neural network architectures, some with explicitly represented linguistic biases and some not, and compare them with respect to their abilities to learn carefully defined linguistic patterns. In contrast to past work that has evaluated linguistic knowledge of neural networks indirectly through a language modeling (word prediction) task, this project instead explores tasks that are formulated as transformations, which map one linguistic form to another (e.g., question formation, verbal inflection, negation, passivization, mapping to logical form). Not only are such mappings at the basis of a widely applied class of neural network architectures, so-called sequence to sequence networks, but they are also a common way of characterizing syntactic processes in linguistics. As a result, the use of such mappings allows a more direct assessment of the networks' linguistic abilities. Part of the project will involve the collaborative development of training and testing datasets for the mappings, with involvement by an interdisciplinary team of linguists and computer scientists, and these will be made available as a resource for the entire research community. These datasets will then be used as the basis for the detailed analysis of neural network representations of linguistic structure. Furthermore, explicit comparisons will be carried out between neural network and human performance on the mapping tasks under study.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.