CRII: RI: Opening the black box of neural natural language processing models using machine-behavioral methods

在我们的日常生活中，我们越来越多地使用自然语言与计算机应用程序交互。该项目的目标是了解和改进支撑这些应用程序的神经网络技术。神经网络是一种人工智能系统，它被训练成以某种方式运行，向它展示了它应该如何运行的许多例子。虽然神经网络是目前已知的构建自然语言应用程序的最佳方式，但它们有一个重大缺陷：一旦它们似乎已经学会了如何做一些自然语言任务，我们就不能确切地知道它们学到了什么，因此我们不能确定它们在所有情况下会如何行动。为了提高这项技术的健壮性，并使其更易于理解和控制，我们需要开发方法来准确确定神经网络在训练后学习了哪些模式。在这个项目中，来自实验心理学的方法被采用并用于揭示神经网络系统的内部工作原理，这些神经网络系统已经被训练来执行自然语言任务。这项研究的重点是确定接受过自然语言处理(NLP)任务训练的神经网络学习自然语言的句法结构的表示有多好：句子中单词之间的语法关系。为此，它采用了最近提出的“机器行为”方法，即通过对神经网络进行行为测试来研究它们。该项目作为一个整体有三个主要组成部分。首先是开发和训练一个大的参考神经网络NLP系统集，这些系统的体系结构各不相同；所有后续的实验都将在所有这些模型上进行，以便可以根据它们表示句法结构的质量对它们进行排名。第二是进行一些行为测试，通过构建精心设计的句子来从神经网络中引发特定的反应。三是使用结构探测这一新技术分析神经网络内部的实际信息，以便直接检测句法结构的表征。这项研究的结果可能有助于指导未来的NLP研究开发更易理解的自然语言应用神经网络系统。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

In our daily lives, we are increasingly interacting with computer applications using natural language. The goal of this project is to understand and improve the neural network technology that underlies these applications. A neural network is an artificial intelligence system that is trained to behave in a certain way by showing it many examples of how it should behave. While neural networks are the best way currently known for building natural language applications, they have a major drawback: Once they appear to have learned how to do some natural language task, we don’t know exactly what they have learned, and consequently we can’t be certain how they will act in all circumstances. In order to improve this technology to be more robust, and also to make it more comprehensible and controllable, we need to develop ways of determining exactly what patterns a neural network has learned once it has been trained. In this project, methods from experimental psychology are adapted and used to reveal the inner workings of neural network systems that have been trained to perform natural language tasks. This research focuses on determining how well neural networks that are trained to do natural language processing (NLP) tasks have learned representations of the syntactic structure of natural language: the grammatical relationships among words in sentences. It does so by adopting the recently proposed “machine behavior” approach in which neural networks are studied by subjecting them to behavioral tests. The project as a whole has three major components. The first is to develop and train a large reference set of neural network NLP systems that vary in their architecture; all subsequent experiments are to be carried out on all of these models, so that they can be ranked in terms of the quality of their representations of syntactic structure. The second is to carry out a number of behavioral tests, by constructing carefully crafted sentences designed to elicit certain responses from the neural networks. The third is to analyze the actual information inside of the neural networks using structural probing, a newly developed technique, in order to detect representations of syntactic structure directly. The results of this research can be useful for guiding future NLP research in the development of more comprehensible neural network systems for natural language applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.