Automated detection of performance regressions using statistical process control techniques
Automated detection of performance regressions using statistical process control techniques
复制标题
使用统计过程控制技术自动检测性能回归
DOI：
--
复制
发表时间：
2012
期刊：
影响因子：
--
通讯作者：
P. Flora
中科院分区：
文献类型：
--
作者：
Thanh H. D. Nguyen;Bram Adams;Z. Jiang;A. Hassan;Mohamed N. Nasser;P. Flora
性能回归测试的目标是检查新版本软件系统中的性能回归。性能回归测试是软件开发过程中的一个重要阶段。性能回归测试非常耗时，但分配给它的时间通常很少。典型的测试运行将输出数千个性能计数器。测试人员通常必须手动检查这些计数器以确定性能倒退。在本文中，我们提出了一种使用称为控制图的统计过程控制技术来分析测试运行中的性能计数器的方法。我们使用一个大型软件团队和一个开源软件项目的历史数据来评估我们的方法。结果表明，我们的方法可以准确地识别两个软件系统中的性能回归。从业者的反馈非常有希望，因为结果很简单，很容易解释。
The goal of performance regression testing is to check for performance regressions in a new version of a software system. Performance regression testing is an important phase in the software development process. Performance regression testing is very time consuming yet there is usually little time assigned for it. A typical test run would output thousands of performance counters. Testers usually have to manually inspect these counters to identify performance regressions. In this paper, we propose an approach to analyze performance counters across test runs using a statistical process control technique called control charts. We evaluate our approach using historical data of a large software team as well as an open-source software project. The results show that our approach can accurately identify performance regressions in both software systems. Feedback from practitioners is very promising due to the simplicity and ease of explanation of the results.