CAREER: Unified Model-agnostic Interpretation Framework for Deep Predictive Models

深度学习模型在各种各样的任务中取得了卓越的预测性能，从计算机视觉到语言处理再到医学图像。许多跨不同领域的组织现在正在构建基于深度学习的大规模应用程序。然而，越来越多的人担心这些模型的公平性和可信度，主要是因为它们的决策过程不透明。例如，当训练的深度学习模型正确地对目标医学图像中的肿瘤进行分类时，必须理解模型学习识别肿瘤的x射线图像部分，以确保发现是有效的。因此，选择和部署值得信赖的深度学习模型迫切需要提供准确合理的解释。该项目将设计和开发一个通用的解释框架，可以应用于各种领域的深度学习应用。解释框架可以对深度神经网络（dnn）感知到的科学知识产生反馈，从而帮助研究人员通过识别、最小化甚至消除不公平和偏见来改进模型。该项目还将在教育活动上投入大量精力，重点关注三个关键领域：(1)K-12教师的专业发展，(2)高中生深度学习夏令营，(3)指导本科生进行研究。这些教育和推广活动将在高中生、K-12教师和大学之间架起桥梁，最终使科学和社会都受益。该项目将开发一种新的解释框架，使人类能够理解在医学图像、视频、自然语言处理和深度强化学习上训练的日益复杂的黑箱dnn的决策过程。尽管在深度神经网络解释方面取得了进展，但上述领域仍存在一些独特的挑战：(1)3D医学图像高度结构化，通常需要领域知识并且难以解释。(2)单纯应用现有的图像解译方法无法实现视频解译。(3)大多数现有的NLP解释模型需要对神经网络的内部结构有一定的了解。(4)目前的DRL解释严重依赖于模仿动作样本的决策树，不能保证最小化政策后悔。本项目将通过以下方式解决这些挑战：(1)通过一种新颖的图形表示来解释3D医学图像，从而在相邻的3D图像切片之间创建相关解释；2)设计基于视频的任务在空间和时间域的显著性估计程序；(3)通过嵌入空间设计一种新的文本摄动方案来识别NLP模型中的重要词；(4)解释agent的行为，阐明agent学习平衡短期和长期奖励的策略；(5)开发了一个包罗万象的解释框架，通过一系列试点应用为任意深度学习模型提供解释。具体的研究任务将在可信度、性能比较和对人类的可解释性方面进行广泛的评估。所有的研究成果将被公开传播，以促进更好地理解可解释的深度神经网络。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning models have achieved exceptional predictive performance in a wide variety of tasks, ranging from computer vision to language processing to medical images. Many organizations across diverse domains are now building large-scale applications based on deep learning. However, there are growing concerns, regarding the fairness and trustworthiness of these models, largely due to the opaque nature of their decision processes. For example, when the trained deep learning model correctly classifies a tumor in a target medical image, the part of the X-ray image that the model learned to identify the tumor in must be understood to ensure the findings are valid. Providing accurate and reasonable interpretations is therefore urgently needed for selecting and deploying trustworthy deep learning models. This project will design and develop a universal interpretation framework that can be applied to a variety of fields for deep learning applications. The interpretation framework can produce feedback on what scientific knowledge is perceived by the Deep Neural Networks (DNNs) and hence helps researchers refine models by identifying, minimizing, or even eliminating unfairness and bias. This project will also spend significant efforts on education activities, focusing on three key areas: (1) professional development for K-12 teachers, (2) deep learning summer camp for high school students, and (3) mentoring undergraduates for research. These educational and outreach activities will build bridges among high school students, K-12 teachers, and colleges that will eventually benefit both science and society. This project will develop a novel interpreting framework that enables humans to understand the decision process of increasing complex black-box DNNs trained on medical images, videos, natural language processing and deep reinforcement learning. Although progress has been achieved on DNN interpretation, several unique challenges remain unexplored for the aforementioned domains: (1) 3D medical images, which are highly structured and usually require domain knowledge and are difficult to explain. (2) Video interpretation cannot be achieved by simply applying existing image interpretation methods. (3) Most existing NLP interpretation models require certain knowledge of the internal structure of the neural networks. (4) Current DRL interpretation heavily relies on decision trees imitating action samples, which cannot guarantee to minimize policy regret. This project will address these challenges in the following ways: (1) Interprets 3D medical images by a novel graphical representation to create correlated interpretations among neighboring slices of 3D images; 2) Devise saliency estimating procedures for video-based tasks in both spatial and temporal domain; (3) Designs a novel text perturbation scheme via embedding space to identify important words of NLP models; (4) Interprets agent's behaviors and elucidates the strategies that agents learn to balance short-term and long-term reward; (5) Develops an all-encompassing interpretation framework to provide interpretations for arbitrary deep learning models through a series of pilot applications. The specific research tasks will be extensively evaluated in trustworthiness, performance comparison, and interpretability to human beings. All the research outcomes will be disseminated publicly to facilitate a better understanding of explainable deep neural networks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.