SHF: Small: AI Model Debugging by Analyzing Model Internals with Python Program Analysis

人工智能（AI），尤其是机器学习（ML），在我们生活的许多方面发挥着越来越重要的作用。AI/ML模型工程（例如，模型训练、调优和维护）因此成为现代软件系统工程的重要组成部分。就像软件不可避免地包含bug，软件调试是软件开发的关键步骤一样，AI/ML模型可能存在bug行为，因此模型调试将是软件工程的关键步骤。人工智能模型的漏洞可能会导致模型精度低、容易受到安全攻击等不良后果，从而极大地阻碍了人工智能模型的应用，尤其是在安全关键领域。目前人工智能模型调试的实践主要集中在参数调优和提供额外的训练数据。但是，它不会尝试从可观察到的症状中诊断根本原因，然后进行相应的修复。没有适用于大量模型的通用调试工具。当前的实践状态是对单个实现执行乏味和冗余的测试，甚至在每个培训会话级别上执行测试。AI模型，尤其是神经网络模型，本质上是通过多个程序阶段（称为层）计算状态变量值（称为神经元激活）的程序（例如，在Python中）。一层神经元的值是通过矩阵乘法和激活函数的应用从前一层的神经元的值计算出来的，激活函数基本上是一个阈值函数，用于确定是否将值用于下一层的计算。该项目将开发将人工智能模型视为具有特定语义的程序的技术，以便人工智能模型调试可以从分析这些程序及其执行状态中受益匪浅。因此，软件工程和程序分析社区在数十年的密集研发中积累的大量软件调试经验可以用来构建通用的和新颖的AI模型调试器。就像软件调试工具帮助开发人员检查程序状态并识别根本原因一样，这些人工智能模型调试器将允许数据工程师比较正确分类和错误分类案例的内部神经元激活值，以识别导致错误分类的根本原因特征，通过提供选择更多输入的指导，以改善与这些特征相关的模型行为。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Artificial Intelligence (AI) in general, and especially Machine Learning (ML), is playing an increasingly important role in many aspects of our life. AI/ML model engineering (e.g., model training, tuning, and maintenance) is hence becoming an essential part of modern software-systems engineering. Just like software inevitably contains bugs and software debugging is a key step in software development, AI/ML models may have buggy behaviors, and thus model debugging will be a critical step in software engineering. AI model bugs may lead to undesirable consequences such as low model accuracy and vulnerabilities to security attacks, which substantially hinder the application of AI models, especially in safety-critical areas. The current practice of AI model debugging mainly focuses on tuning parameters and providing additional training data. However, it does not try to diagnose the root cause from the observable symptoms and then repair accordingly. There are no general debugging tools that work for a large set of models. The current state of practice is to perform tedious and redundant tests on individual implementations and even at a per-training-session level. AI models, especially neural network models, are essentially programs (e.g., in Python) that compute state-variable values, called neuron activations, through multiple program phases (called layers). The values of neurons in a layer are computed from those of the previous layer through matrix multiplication and application of an activation function, which is basically a threshholding function to determine if values will be used in the computation of the next layer. The project will develop techniques that consider AI models as programs with specific semantics such that AI model debugging can substantially benefit from analyzing these programs and their execution states. As such, the substantial experience of software debugging that is built up by the software-engineering and program-analysis community over decades of intensive R&D can be leveraged to build general and novel AI model debuggers. Just like software-debugging tools help developers to inspect program states and identify root causes, these AI model debuggers will allow data engineers to compare internal neuron-activation values of correctly classified and mis-classified cases to identify root cause features that lead to misclassification, by providing guidance in selecting more inputs to improve model behavior related to these features.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.