Collaborative Research: RI: Small: Post hoc Explanations in the Wild: Exposing Vulnerabilities and Ensuring Robustness

在医疗保健和刑事司法等关键领域成功采用机器学习（ML）模型，在很大程度上取决于决策者对这些模型功能的理解和信任程度。然而，机器学习模型的专有性质和日益增加的复杂性使得领域专家很难理解这些复杂的“黑盒子”。因此，最近出现了大量的技术，这些技术通过使用更简单的模型来近似地以人类可解释的方式解释黑盒模型。然而，目前还不清楚这些事后解释技术会在多大程度上误导最终用户，给他们一种虚假的安全感，并诱使他们信任和部署不可信的黑盒。该项目将构建严格的框架来暴露现有解释技术的漏洞，评估这些漏洞如何在现实世界的应用程序中表现出来，并开发新的技术来防御这些漏洞。该项目有可能显著加快ML在各种领域的应用，包括刑事司法（例如，保释决定）、医疗保健（例如，患者诊断和治疗）和金融贷款（例如，贷款批准）。该项目的目标是描述现有解释技术的漏洞，了解攻击者如何利用这些漏洞，并开发防御它们的技术。该项目将侧重于以下子任务：1)通过对用户进行研究，并与医疗保健和刑事司法领域的专家进行详细访谈，了解误导性解释在现实世界中的后果；2)识别最先进的解释技术中的关键漏洞，这些漏洞可能被敌对实体利用，产生误导性解释；3)开发新的技术来构建健壮和可靠的解释，这些解释不容易受到这些漏洞的影响，从而为领域专家和其他利益相关者提供复杂黑箱模型的忠实解释。有了这些贡献，该项目将启动一个新的机器学习可解释性研究体系，重点是了解对手如何操纵解释技术，以及如何防御此类攻击。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The successful adoption of machine learning (ML) models in critical domains such as healthcare and criminal justice relies heavily on how well decision makers are able to understand and trust the functionality of these models. However, the proprietary nature and increasing complexity of ML models makes it challenging for domain experts to understand these complex "black boxes". Consequently, there has been a recent surge in techniques that explain black box models in a human interpretable manner by approximating them using simpler models. However, it is unclear to what extent these post hoc explanation techniques may mislead end users by giving them a false sense of security, and luring them into trusting and deploying untrustworthy black boxes. This project will build rigorous frameworks to expose the vulnerabilities of existing explanation techniques, assess how these vulnerabilities can manifest in real world applications, and develop new techniques to defend against these vulnerabilities. This project has the potential to significantly speed up the adoption of ML in a variety of domains including criminal justice (e.g., bail decisions), health care (e.g., patient diagnosis and treatment), and financial lending (e.g., loan approval).The goal of this project is to characterize the vulnerabilities of existing explanation techniques, understand how adversaries can exploit these vulnerabilities, and develop techniques to defend against them. The project will focus on the following subtasks: 1) understanding the real-world consequences of misleading explanations by conducting user studies and detailed interviews with domain experts in healthcare and criminal justice 2) identifying critical vulnerabilities in state-of-the-art explanation techniques that can be exploited by adversarial entities to generate misleading explanations, and 3) developing novel techniques for building robust and reliable explanations that are not prone to these vulnerabilities and thereby provide domain experts and other stakeholders with faithful explanations of complex black box models. With these contributions, the project will initiate a new body of research in ML interpretability that focuses on understanding how adversaries can manipulate explanation techniques, and how to defend against such attacks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.