CRII: RI: Opening the black box of neural natural language processing models using machine-behavioral methods

点击翻译按钮获取中文摘要

In our daily lives, we are increasingly interacting with computer applications using natural language. The goal of this project is to understand and improve the neural network technology that underlies these applications. A neural network is an artificial intelligence system that is trained to behave in a certain way by showing it many examples of how it should behave. While neural networks are the best way currently known for building natural language applications, they have a major drawback: Once they appear to have learned how to do some natural language task, we don’t know exactly what they have learned, and consequently we can’t be certain how they will act in all circumstances. In order to improve this technology to be more robust, and also to make it more comprehensible and controllable, we need to develop ways of determining exactly what patterns a neural network has learned once it has been trained. In this project, methods from experimental psychology are adapted and used to reveal the inner workings of neural network systems that have been trained to perform natural language tasks. This research focuses on determining how well neural networks that are trained to do natural language processing (NLP) tasks have learned representations of the syntactic structure of natural language: the grammatical relationships among words in sentences. It does so by adopting the recently proposed “machine behavior” approach in which neural networks are studied by subjecting them to behavioral tests. The project as a whole has three major components. The first is to develop and train a large reference set of neural network NLP systems that vary in their architecture; all subsequent experiments are to be carried out on all of these models, so that they can be ranked in terms of the quality of their representations of syntactic structure. The second is to carry out a number of behavioral tests, by constructing carefully crafted sentences designed to elicit certain responses from the neural networks. The third is to analyze the actual information inside of the neural networks using structural probing, a newly developed technique, in order to detect representations of syntactic structure directly. The results of this research can be useful for guiding future NLP research in the development of more comprehensible neural network systems for natural language applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.