Reflection-aware static regression test selection
Reflection-aware static regression test selection
复制标题
DOI：
10.1145/3360613
复制
发表时间：
2019-10
影响因子：
--
通讯作者：
A. Shi;Milica Hadzi-Tanovic;Lingming Zhang;D. Marinov;Owolabi Legunsen
中科院分区：
文献类型：
--
作者：
A. Shi;Milica Hadzi-Tanovic;Lingming Zhang;D. Marinov;Owolabi Legunsen
回归测试选择(RTS)旨在通过仅重新运行受代码更改影响的测试来加速回归测试。可以使用静态或动态分析技术来执行RTS。我们之前的研究表明，对于中等规模的Java项目，静态和动态RTS的性能相似。然而，先前研究的结果也表明，静态RTS可能是不安全的，没有选择动态RTS选择的测试，而反射是在被评估的项目中观察到的不安全的唯一原因。在本文中，我们研究了五种技术--三种纯静态技术和两种静态-动态混合技术--旨在使静态RTS相对于反射而言是安全的。我们通过在名为Start的工具中扩展反射不感知(RU)类级别的静态RTS技术来实现这些反射感知(RA)技术。为了评估这些RA技术，我们将它们的端到端时间与RU和RetestAll进行了比较，RetestAll在每次代码更改后重新运行所有测试。我们还比较了RA技术和Ekstazi技术的安全性和精确度，Ekstazi是一种最先进的动态RTS技术；精确度是所选择的不受影响的测试的衡量标准。我们对24个开源Java项目的1173个版本的评估结果是否定的。RA技术提高了RU的安全性，但成本非常高。纯静态技术在我们的实验中是安全的，但降低了RU的精度，端到端时间最多为RetestAll时间的85.8%，而RU为69.1%。一种静态-动态混合技术提高了RU的安全性，但代价很高，端到端时间是RetestAll的91.2%。另一种混合静态-动态技术提供了更好的精度，比RU更安全，并且产生了更低的端到端时间--RetestAll的75.8%，但在存在测试顺序依赖的情况下，它仍然是不安全的。我们的研究强调了使静态RTS相对于反射安全所涉及的挑战。
Regression test selection (RTS) aims to speed up regression testing by rerunning only tests that are affected by code changes. RTS can be performed using static or dynamic analysis techniques. Our prior study showed that static and dynamic RTS perform similarly for medium-sized Java projects. However, the results of that prior study also showed that static RTS can be unsafe, missing to select tests that dynamic RTS selects, and that reflection was the only cause of unsafety observed among the evaluated projects. In this paper, we investigate five techniques—three purely static techniques and two hybrid static-dynamic techniques—that aim to make static RTS safe with respect to reflection. We implement these reflection-aware (RA) techniques by extending the reflection-unaware (RU) class-level static RTS technique in a tool called STARTS. To evaluate these RA techniques, we compare their end-to-end times with RU, and with RetestAll, which reruns all tests after every code change. We also compare safety and precision of the RA techniques with Ekstazi, a state-of-the-art dynamic RTS technique; precision is a measure of unaffected tests selected. Our evaluation on 1173 versions of 24 open-source Java projects shows negative results. The RA techniques improve the safety of RU but at very high costs. The purely static techniques are safe in our experiments but decrease the precision of RU, with end-to-end time at best 85.8% of RetestAll time, versus 69.1% for RU. One hybrid static-dynamic technique improves the safety of RU but at high cost, with end-to-end time that is 91.2% of RetestAll. The other hybrid static-dynamic technique provides better precision, is safer than RU, and incurs lower end-to-end time—75.8% of RetestAll, but it can still be unsafe in the presence of test-order dependencies. Our study highlights the challenges involved in making static RTS safe with respect to reflection.