SHF: Small: AI Model Debugging by Analyzing Model Internals with Python Program Analysis

总的来说，人工智能(AI)，特别是机器学习(ML)，在我们生活的许多方面发挥着越来越重要的作用。因此，AI/ML模型工程(例如，模型培训、调优和维护)正在成为现代软件系统工程的重要组成部分。正如软件不可避免地存在缺陷一样，软件调试是软件开发的关键步骤，AI/ML模型可能存在缺陷行为，因此模型调试将是软件工程中的关键步骤。AI模型缺陷可能会导致模型准确率低、安全攻击漏洞等不良后果，这大大阻碍了AI模型的应用，特别是在安全关键领域。目前AI模型调试的实践主要集中在调整参数和提供额外的训练数据上。然而，它并不试图根据可观察到的症状来诊断根本原因，然后进行相应的修复。没有适用于大型模型集的通用调试工具。目前的实践状态是对单个实现执行乏味和冗余的测试，甚至在每个培训会话的级别上执行。人工智能模型，特别是神经网络模型，本质上是通过多个程序阶段(称为层)计算状态变量值的程序(例如，在Python语言中)，称为神经元激活。一层神经元的值是通过矩阵相乘和应用激活函数从上一层神经元的值计算出来的，该激活函数基本上是一个阈值函数，用于确定下一层的值是否将用于计算。该项目将开发将AI模型视为具有特定语义的程序的技术，以便AI模型调试可以从分析这些程序及其执行状态中获得实质性好处。因此，软件工程和程序分析社区在数十年的密集研发过程中积累的丰富软件调试经验可以用来构建通用的、新颖的AI模型调试器。就像软件调试工具帮助开发人员检查程序状态并确定根本原因一样，这些AI模型调试器将允许数据工程师比较正确分类和错误分类案例的内部神经元激活值，通过在选择更多输入来改善与这些特征相关的模型行为方面提供指导，从而识别导致错误分类的根本原因特征。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Artificial Intelligence (AI) in general, and especially Machine Learning (ML), is playing an increasingly important role in many aspects of our life. AI/ML model engineering (e.g., model training, tuning, and maintenance) is hence becoming an essential part of modern software-systems engineering. Just like software inevitably contains bugs and software debugging is a key step in software development, AI/ML models may have buggy behaviors, and thus model debugging will be a critical step in software engineering. AI model bugs may lead to undesirable consequences such as low model accuracy and vulnerabilities to security attacks, which substantially hinder the application of AI models, especially in safety-critical areas. The current practice of AI model debugging mainly focuses on tuning parameters and providing additional training data. However, it does not try to diagnose the root cause from the observable symptoms and then repair accordingly. There are no general debugging tools that work for a large set of models. The current state of practice is to perform tedious and redundant tests on individual implementations and even at a per-training-session level. AI models, especially neural network models, are essentially programs (e.g., in Python) that compute state-variable values, called neuron activations, through multiple program phases (called layers). The values of neurons in a layer are computed from those of the previous layer through matrix multiplication and application of an activation function, which is basically a threshholding function to determine if values will be used in the computation of the next layer. The project will develop techniques that consider AI models as programs with specific semantics such that AI model debugging can substantially benefit from analyzing these programs and their execution states. As such, the substantial experience of software debugging that is built up by the software-engineering and program-analysis community over decades of intensive R&D can be leveraged to build general and novel AI model debuggers. Just like software-debugging tools help developers to inspect program states and identify root causes, these AI model debuggers will allow data engineers to compare internal neuron-activation values of correctly classified and mis-classified cases to identify root cause features that lead to misclassification, by providing guidance in selecting more inputs to improve model behavior related to these features.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.