Fine-grained Interpretation of Deep Neural Network Models of NLP

我们解释深度神经网络（dnn）的能力是迈向可解释人工智能的重要一步，可以确保模型的公平性，并实现更好的泛化学习模型。在本提案中，我们致力于通过提供对其表示的细粒度解释来提高DNN模型的透明度。细粒度解释的目标是特定知识如何在网络中编码。它回答了一些问题，比如，神经元在神经网络中的作用是什么？什么任务和语言知识是在神经元中学习的？了解神经元的作用及其在网络中的重要性，为建立透明、公平、可解释和高效的模型铺平了道路。在本提案中，我们将对dnn的细粒度解释做出多项贡献。更具体地说，我们将提出解释的数据集和算法。我们将致力于解释模型预测的因果解释。此外，我们将开发由神经元分析实现的应用程序。具体项目总结如下：概念数据集：目前DNN解释的工作是研究人类定义的语言概念（如名词、动词等）是否在模型中被学习。然而，它们不分析模型已经了解的关于任务和语言的其他潜在概念。该项目旨在准备一个以模型为中心的概念数据集。我们将使用上下文化表示来识别高维空间中的一组单词，并在一定的标准下手动注释它们。所得到的数据集是一个新颖的多方面数据，并将实现以模型为中心的解释。解释算法：神经元本质上是多元的，由于训练和架构的选择，模型表现出冗余。如何识别冗余学习概念的神经元？如何找到一组一起工作的神经元来代表一个概念？常用的神经元分析方法在很大程度上忽略了这些问题，或者需要仔细评估才能证明其有效性。本项目旨在改进神经元分析方法。我们将探索各种多元特征选择方法，如（稀疏）组套索来分析神经元。因果解释：概念对于类别或特定预测的重要性对于可解释的AI至关重要。这个项目的目标是基于概念的预测因果解释。我们将使用梯度和基于微扰的归因算法来识别预测的显著神经元，并将其与神经元分析联系起来，以生成基于概念的预测解释。应用：在这个项目中，我们将探索神经元分析的应用，更具体地说，是控制模型的行为和领域适应。其想法是根据感兴趣的概念（如性别或领域）识别神经元，并开发在测试时操纵这些神经元的方法，以便以理想的方式控制模型的行为。

Our ability to interpret Deep neural networks (DNNs) is an essential step towards explainable AI, to ensure fairness of models, and in achieving models that learn better generalization. In this proposal, we work towards increasing the transparency of DNN models by providing fine-grained interpretation of their representation. Fine-grained interpretation targets how certain knowledge is encoded in the network. It answers questions such as, what is the role of neurons in the network? What task and language knowledge is learned in neurons?, Knowing the role of neurons and their importance in the network paves the way towards transparent, fair, explainable and efficient models. In this proposal, we will make multiple contributions to fine-grained interpretation of DNNs. More specifically, we will propose datasets and algorithms for interpretation. We will work towards causal interpretation that explains the prediction of a model. Moreover, we will develop applications that are enabled by neuron analysis. The specific projects are summarized below: Concept Dataset: Current work in DNN interpretation probes whether human-defined linguistic concepts such as noun, verb, etc. are learned in a model. However, they do not analyze what other latent concepts a model has learned about the task and the language. This project aims at preparing a model-centric concept dataset. We will identify group of words in high dimensional space using their contextualized representation and will manually annotate them under certain criteria. The resulting dataset is a novel multi-facet data and will enable model-centered interpretation. Interpretation Algorithms: Neurons are multivariate in nature and models exhibit redundancy due to training and architectural choices. How to identify neurons that redundantly learn a concept? How to find a group of neurons that work together to represent a concept? Common approaches to neuron analysis largely ignore these questions or require careful evaluation to prove their efficacy. This project aims at improving methods of neuron analysis. We will explore various multivariate feature selection methods such as (sparse) group lasso to analyze neurons. Causal Interpretation: The importance of a concept for a class or for a specific prediction is essential for explainable AI. This project targets concept-based causal interpretation of prediction. We will use gradient and perturbation-based attribution algorithms to identify salient neurons for a prediction and will connect it with neuron analysis to generate concept-based explanations of predictions. Applications: In this project, we will explore applications of neuron analysis, more specifically controlling model's behavior and domain adaptation. The idea is to identify neurons with respect to concepts of interest e.g. gender or domain and develop methods to manipulate those neurons at test time in order to control the behavior of the model in a desirable way.