An empirical assessment of machine learning approaches for triaging reports of static analysis tools
An empirical assessment of machine learning approaches for triaging reports of static analysis tools
复制标题
DOI：
10.1007/s10664-022-10253-z
复制
发表时间：
2023-03-01
影响因子：
4.1
通讯作者：
Porter，Adam A.
中科院分区：
文献类型：
--
作者：
Yerramreddy，Sai;Mordahl，Austin;Porter，Adam A.
尽管静态分析工具能够检测软件中的关键错误，但其高误报率是其在现实环境中应用的主要障碍。为了提高这些工具的可用性，研究人员最近开始应用机器学习技术来分类和过滤不正确的分析报告。虽然最初的结果很有希望，但由于缺乏详细的、大规模的实证评估，这条研究路线的长期潜力和最佳实践尚不清楚。为了部分解决这一知识差距，我们提出了三种机器学习技术的比较实证研究-传统模型，循环神经网络（rnn）和图神经网络(gnn) -用于在三种静态分析工具（findsecbugs， CBMC和jbmc）中分类正确和不正确的结果-使用多个数据集。这些工具代表了不同的静态分析技术，即污染分析和模型检查。我们还介绍并评估了用于rnn的新的数据准备例程和用于gnn的节点表示。我们发现，对于不同的数据集和应用场景，总体分类准确率达到80%-99%。我们观察到数据准备例程对分类精度有积极的影响，rnn和gnn的分类精度分别提高了5%和16%。总体而言，我们的研究结果表明，通过程序源代码学习的神经网络（rnn或gnn）优于传统模型，尽管所有技术之间存在有趣的权衡。我们的观察为未来的研究提供了洞察力，这些研究需要在实践中加速采用静态分析工具的机器学习方法。
Despite their ability to detect critical bugs in software, static analysis tools’ high false positive rates are a key barrier to their adoption in real-world settings. To improve the usability of these tools, researchers have recently begun to apply machine learning techniques to classify and filter incorrect analysis reports. Although initial results have been promising, the long-term potential and best practices for this line of research are unclear due to the lack of detailed, large-scale empirical evaluation. To partially address this knowledge gap, we present a comparative empirical study of three machine learning techniques—traditional models, recurrent neural networks (RNNs), and graph neural networks (GNNs)—for classifying correct and incorrect results in three static analysis tools—FindSecBugs, CBMC, and JBMC—using multiple datasets. These tools represent different techniques of static analysis, namely taint analysis and model-checking. We also introduce and evaluate new data preparation routines for RNNs and node representations for GNNs. We find that overall classification accuracy reaches a high of 80%–99% for different datasets and application scenarios. We observe that data preparation routines have a positive impact on classification accuracy, with an improvement of up to 5% for RNNs and 16% for GNNs. Overall, our results suggest that neural networks (RNNs or GNNs) that learn over a program’s source code outperform traditional models, although interesting tradeoffs are present among all techniques. Our observations provide insight into the future research needed to speed the adoption of machine learning approaches for static analysis tools in practice.