CAREER: Unified Model-agnostic Interpretation Framework for Deep Predictive Models

深度学习模型在从计算机视觉到语言处理再到医学图像的各种任务中都取得了出色的预测性能。许多不同领域的组织现在都在构建基于深度学习的大规模应用程序。然而，人们越来越担心这些模型的公平性和可信度，这在很大程度上是因为它们的决策过程不透明。例如，当训练的深度学习模型正确地对目标医学图像中的肿瘤进行分类时，必须理解模型学习识别肿瘤的X射线图像部分，以确保发现是有效的。因此，为选择和部署值得信赖的深度学习模型，迫切需要提供准确和合理的解释。这个项目将设计和开发一个通用的口译框架，可以应用于深度学习的各种领域。解释框架可以对深度神经网络(DNN)感知的科学知识产生反馈，从而帮助研究人员通过识别、最小化甚至消除不公平和偏见来改进模型。该项目还将在教育活动上投入大量精力，重点放在三个关键领域：(1)K-12教师的专业发展，(2)高中生深度学习夏令营，(3)指导本科生进行研究。这些教育和外展活动将在高中生、K-12教师和大学之间架起桥梁，最终将使科学和社会受益。该项目将开发一种新的口译框架，使人类能够理解在医学图像、视频、自然语言处理和深度强化学习方面训练的越来越复杂的黑盒DNN的决策过程。虽然在DNN解释方面已经取得了进展，但在上述领域中仍有几个独特的挑战尚未探索：(1)3D医学图像，它高度结构化，通常需要领域知识，并且很难解释。(2)简单地应用现有的图像判读方法无法实现视频判读。(3)大多数现有的自然语言处理解释模型都需要了解神经网络的内部结构。(4)当前的DRL解释严重依赖于模拟动作样本的决策树，这不能保证最大限度地减少政策遗憾。本项目将通过以下方式解决这些挑战：(1)通过一种新的图形表示来解释3D医学图像，以在3D图像的相邻切片之间创建相关解释；(2)设计基于视频的任务在空间和时间域的显著估计程序；(3)设计一种新的文本扰动方案，通过嵌入空间来识别NLP模型的重要单词；(4)解释代理的行为并阐明代理学习如何平衡短期和长期回报的策略；(5)开发一个全面的解释框架，通过一系列试点应用为任意深度学习模型提供解释。具体的研究任务将在可信性、性能比较和对人类的可解释性方面进行广泛的评估。所有研究成果将被公开传播，以促进对可解释的深层神经网络的更好理解。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning models have achieved exceptional predictive performance in a wide variety of tasks, ranging from computer vision to language processing to medical images. Many organizations across diverse domains are now building large-scale applications based on deep learning. However, there are growing concerns, regarding the fairness and trustworthiness of these models, largely due to the opaque nature of their decision processes. For example, when the trained deep learning model correctly classifies a tumor in a target medical image, the part of the X-ray image that the model learned to identify the tumor in must be understood to ensure the findings are valid. Providing accurate and reasonable interpretations is therefore urgently needed for selecting and deploying trustworthy deep learning models. This project will design and develop a universal interpretation framework that can be applied to a variety of fields for deep learning applications. The interpretation framework can produce feedback on what scientific knowledge is perceived by the Deep Neural Networks (DNNs) and hence helps researchers refine models by identifying, minimizing, or even eliminating unfairness and bias. This project will also spend significant efforts on education activities, focusing on three key areas: (1) professional development for K-12 teachers, (2) deep learning summer camp for high school students, and (3) mentoring undergraduates for research. These educational and outreach activities will build bridges among high school students, K-12 teachers, and colleges that will eventually benefit both science and society. This project will develop a novel interpreting framework that enables humans to understand the decision process of increasing complex black-box DNNs trained on medical images, videos, natural language processing and deep reinforcement learning. Although progress has been achieved on DNN interpretation, several unique challenges remain unexplored for the aforementioned domains: (1) 3D medical images, which are highly structured and usually require domain knowledge and are difficult to explain. (2) Video interpretation cannot be achieved by simply applying existing image interpretation methods. (3) Most existing NLP interpretation models require certain knowledge of the internal structure of the neural networks. (4) Current DRL interpretation heavily relies on decision trees imitating action samples, which cannot guarantee to minimize policy regret. This project will address these challenges in the following ways: (1) Interprets 3D medical images by a novel graphical representation to create correlated interpretations among neighboring slices of 3D images; 2) Devise saliency estimating procedures for video-based tasks in both spatial and temporal domain; (3) Designs a novel text perturbation scheme via embedding space to identify important words of NLP models; (4) Interprets agent's behaviors and elucidates the strategies that agents learn to balance short-term and long-term reward; (5) Develops an all-encompassing interpretation framework to provide interpretations for arbitrary deep learning models through a series of pilot applications. The specific research tasks will be extensively evaluated in trustworthiness, performance comparison, and interpretability to human beings. All the research outcomes will be disseminated publicly to facilitate a better understanding of explainable deep neural networks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.