An Initial Analysis of Repair and Side-effect Prediction for Neural Networks
An Initial Analysis of Repair and Side-effect Prediction for Neural Networks
复制标题
DOI：
10.1109/cain58948.2023.00017
复制
发表时间：
2023-05
期刊：
影响因子：
--
通讯作者：
Yuta Ishimoto;Ken Matsui;Masanari Kondo;Naoyasu Ubayashi;Yasutaka Kamei
中科院分区：
文献类型：
--
作者：
Yuta Ishimoto;Ken Matsui;Masanari Kondo;Naoyasu Ubayashi;Yasutaka Kamei
随着采用神经网络模型的软件系统的流行，这些系统的质量保证变得至关重要。因此，各种研究提出了神经网络模型的修复方法，以提高模型的质量。虽然研究人员对这些方法进行了评估，但很难判断它们是否在所有模型和数据集中都取得了成功（即，所有开发人员的环境）。由于这些方法需要许多资源，例如执行时间，因此无法修复神经网络将使开发人员花费他们的资源。因此，如果开发人员能够在采用修复方法之前知道它们是否成功，他们可以避免浪费资源。本文提出了预测模型，预测修复方法是否成功地修复神经网络使用一个小的资源。我们的预测模型分别预测修复方法的修复和副作用。我们在三个数据集Fashion-MNIST、CIFAR-10和GTSRB上评估了我们的预测模型，发现我们的预测模型具有很高的性能，副作用的平均ROC-AUC为0.931，平均f1得分为0.880，修复的平均ROC-AUC为0.768，平均f1得分为0.725。
With the prevalence of software systems adopting neural network models, the quality assurance of these systems has become crucial. Hence, various studies have proposed repairing methods for neural network models so far to improve the quality of the models. While these methods are evaluated by researchers, it is difficult to tell whether they succeed in all models and datasets (i.e., all developers’ environments). Because these methods require many resources, such as execution times, failing to repair neural networks would cost developers their resources. Hence, if developers can know whether repairing methods succeed before adopting them, they could avoid wasting their resources. This paper proposes prediction models that predict whether repairing methods succeed in repairing neural networks using a small resource. Our prediction models predict repairs and side-effects of repairing methods, respectively. We evaluated our prediction models on a state-of-the-art repairing method Arachne on three datasets, Fashion-MNIST, CIFAR-10, and GTSRB, and found our prediction models achieved high performance, an average ROC-AUC of 0.931 and an average f1score of 0.880 for the side-effects and an average ROC-AUC of 0.768 and an average f1-score of 0.725 for the repairs.