CRII: RI: Opening the black box of neural natural language processing models using machine-behavioral methods

在我们的日常生活中，我们越来越多地使用自然语言与计算机应用程序进行交互。这个项目的目标是理解和改进这些应用背后的神经网络技术。神经网络是一种人工智能系统，通过向它展示许多应该如何表现的例子来训练它以某种方式表现。虽然神经网络是目前已知的构建自然语言应用程序的最佳方式，但它们有一个主要缺点：一旦它们看起来已经学会了如何完成一些自然语言任务，我们就无法确切地知道它们学了什么，因此我们无法确定它们在所有情况下会如何行动。为了改进这项技术，使其更健壮，也使其更易于理解和控制，我们需要开发出一种方法，来确定神经网络在接受训练后究竟学到了什么模式。在这个项目中，实验心理学的方法被用来揭示神经网络系统的内部工作原理，这些神经网络系统已经被训练来执行自然语言任务。这项研究的重点是确定被训练来完成自然语言处理（NLP）任务的神经网络如何很好地学习自然语言的句法结构表征：句子中单词之间的语法关系。它采用了最近提出的“机器行为”方法，通过对神经网络进行行为测试来研究神经网络。整个项目有三个主要组成部分。首先是开发和训练一个大型的参考神经网络NLP系统，这些系统的架构各不相同；所有后续的实验都将在所有这些模型上进行，以便根据它们对句法结构的表示质量对它们进行排名。第二种方法是进行一系列行为测试，通过构建精心设计的句子来引发神经网络的某些反应。三是利用结构探测技术对神经网络内部的实际信息进行分析，从而直接检测句法结构的表征。本研究的结果可以用于指导未来的NLP研究，以开发更易于理解的自然语言应用的神经网络系统。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

In our daily lives, we are increasingly interacting with computer applications using natural language. The goal of this project is to understand and improve the neural network technology that underlies these applications. A neural network is an artificial intelligence system that is trained to behave in a certain way by showing it many examples of how it should behave. While neural networks are the best way currently known for building natural language applications, they have a major drawback: Once they appear to have learned how to do some natural language task, we don’t know exactly what they have learned, and consequently we can’t be certain how they will act in all circumstances. In order to improve this technology to be more robust, and also to make it more comprehensible and controllable, we need to develop ways of determining exactly what patterns a neural network has learned once it has been trained. In this project, methods from experimental psychology are adapted and used to reveal the inner workings of neural network systems that have been trained to perform natural language tasks. This research focuses on determining how well neural networks that are trained to do natural language processing (NLP) tasks have learned representations of the syntactic structure of natural language: the grammatical relationships among words in sentences. It does so by adopting the recently proposed “machine behavior” approach in which neural networks are studied by subjecting them to behavioral tests. The project as a whole has three major components. The first is to develop and train a large reference set of neural network NLP systems that vary in their architecture; all subsequent experiments are to be carried out on all of these models, so that they can be ranked in terms of the quality of their representations of syntactic structure. The second is to carry out a number of behavioral tests, by constructing carefully crafted sentences designed to elicit certain responses from the neural networks. The third is to analyze the actual information inside of the neural networks using structural probing, a newly developed technique, in order to detect representations of syntactic structure directly. The results of this research can be useful for guiding future NLP research in the development of more comprehensible neural network systems for natural language applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.