CRII: RI: Using Linguistic Variation to Understand Deep Neural Models of Language

许多成功的现代计算语言系统都依赖于深度神经网络。虽然旧的技术依赖于结构化的语言表示，但神经模型的内部工作即使对创建它们的工程师和科学家也是不透明的。因此，与人工智能的其他领域一样，自然语言处理目前面临的一个主要挑战是开发方法，使我们能够理解不透明神经模型的内部表示。自然语言处理是唯一准备为这一奋进做出贡献的原因有两个。首先，语言学领域长期以来一直在寻求开发工具来描述处理人类语言所需的各种表征，因此有大量的前期工作可以借鉴;其次，世界语言的广度和多样性为我们提供了一种自然的方法来研究不同但同样有效的参数化模型。正如研究人类如何处理不同的语言可以深入了解人类语言处理和人类认知一样，了解多语言计算系统如何处理不同的语言可以深入了解计算模型。其中一些模型是大量多语言的，可以同时使用大量语言进行训练。有趣的是，这些多语言模型似乎同时获得了特定语言和通用语言的知识。利用语言技术和世界语言之间的变化，这个计算机研究启动研究（CRII）项目进行了一系列计算实验，涉及在大规模多语言模型的预训练嵌入空间上训练小分类器（例如，多语言BERT和XLM-Roberta），并使用分类器输出来表征这些模型如何表示语言的关键语法方面（例如，语法主语）跨不同形态句法系统的语言。此外，为了开发更强大的方法来研究这些模型中的语法角色，该项目使用计算技术来构建并公开发布更丰富的注释多语言语料库。实验结果和公共语料库有助于我们理解计算语言模型，并使使用这些技术可以研究的语言集多样化。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Many successful modern computational language systems rely on deep neural networks. Whereas older techniques relied on structured linguistic representations, the inner workings of neural models can be opaque even to the engineers and scientists who create them. Therefore, a current major challenge in Natural Language Processing, as in other areas of Artificial Intelligence, is to develop methods that allow us to understand the internal representations of opaque neural models. Natural Language Processing is uniquely poised to contribute to this endeavor for two reasons. First, the field of linguistics has long sought to develop tools for characterizing the kinds of representations necessary for processing human language, and so there is a rich body of prior work to draw on. Second, the breadth and variation of world languages give us a natural way of studying models under different, but equally valid, parameterizations. Just as studying how humans can process diverse languages gives insight into human language processing and human cognition, understanding how multilingual computational systems process different languages can give insights into computational models.A class of deep neural models, known as transformers, has been particularly successful at natural language tasks. Some of these models are massively multilingual, trained on large numbers of languages at once. Interestingly, these multilingual models seem to acquire both language-specific and language-general knowledge. Taking advantage of linguistic techniques and variation among world languages, this Computer Research Initiation Research (CRII) project undertakes a series of computational experiments that involve training small classifiers on the pre-trained embedding space of massive multilingual models (e.g., Multilingual BERT and XLM-Roberta) and using the classifier output to characterize how these models represent crucial grammatical aspects of language (e.g., grammatical subject) across languages with different morphosyntactic systems. Moreover, in order to develop more robust ways of studying grammatical roles in these models, the project uses computational techniques to build and publicly release more richly annotated multilingual corpora. The experimental results and public corpora contribute both to our understanding of computational language models and diversify the set of languages that can be studied using these techniques.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.