Stability of Product-Line Samplingin Continuous Integration
Stability of Product-Line Samplingin Continuous Integration
复制标题
DOI：
10.1145/3442391.3442410
复制
发表时间：
2021-02
期刊：
影响因子：
--
通讯作者：
T. Pett;S. Krieter;Tobias Runge;Thomas Thüm;Malte Lochau;Ina Schaefer
中科院分区：
文献类型：
--
作者：
T. Pett;S. Krieter;Tobias Runge;Thomas Thüm;Malte Lochau;Ina Schaefer
公司努力在其开发过程中实施持续集成，以确保其系统的质量。CI过程中的回归测试考虑了更改后系统的有效重新测试。然而，即使使用回归测试，由于组合爆炸问题，从高度可配置的软件系统测试所有配置也是不可行的。已经提出了许多采样算法，其目的是计算要测试的相当小但足够具有代表性的配置集。这些算法通常根据效率(即，样本中的配置数量和用于生成样本的计算工作量)和有效性(即，特征交互覆盖范围或检测到的故障数量)进行评估。在这篇文章中，我们认为，采样算法的另一个重要特征是，当它们连续应用于演化的可配置系统时，它们倾向于产生相似的配置。我们提出了采样稳定性作为采样算法的一个新的评价标准。给出了一种基于连续样本之间的相似性来计算采样算法的采样稳定性的方法。在我们的评估中，我们比较了多个已建立的t向采样算法在大型真实系统上的采样稳定性。
Companies strive to implement continuous integration into their development process to ensure the quality of their systems. Regression testing within the CI process considers the efficient re-test of systems after changes. However, even with regression testing, it is not feasible to test all configurations from a highly-configurable software system due to the combinatorial-explosion problem. Numerous sampling algorithms have been proposed that aim at computing a considerably smaller yet sufficiently representative set of configurations to be tested. Those algorithms are typically evaluated with regard to efficiency (i.e., number of configurations in a sample and computational effort for generating a sample) and effectiveness (i.e., feature-interaction coverage or number of faults detected). In this paper, we argue that a further crucial characteristic of sampling algorithms is their tendency to produce similar configurations when applied consecutively to an evolving configurable system. We propose sampling stability as a new evaluation criterion for sampling algorithms. We present a procedure to compute the sampling stability of sampling algorithms based on the similarity between consecutive samples. In our evaluation, we compare the sampling stability of multiple established t-wise sampling algorithms on large real-world systems.