How effective are existing Java API specifications for finding bugs during runtime verification?
How effective are existing Java API specifications for finding bugs during runtime verification?
复制标题
DOI：
10.1007/s10515-019-00267-1
复制
发表时间：
2019-11
影响因子：
3.4
通讯作者：
Owolabi Legunsen;Nader Al Awar;Xinyue Xu;Wajih Ul Hassan;Grigore Roşu;D. Marinov
中科院分区：
文献类型：
--
作者：
Owolabi Legunsen;Nader Al Awar;Xinyue Xu;Wajih Ul Hassan;Grigore Roşu;D. Marinov
在软件开发过程中，通过根据正式规范（specs）监视测试执行，可使用验证来早期发现错误。运行时验证的质量取决于规范的质量。虽然以前的研究已经为Java API产生了许多规范，手动或通过自动挖掘，但还没有大规模的研究它们的bug查找效率。我们的会议论文提出了对以前提出的规范的缺陷发现有效性的第一次深入研究。我们使用JavaMOP监控了200个开源项目中的18000多个手动编写的测试方法和210万个自动生成的测试方法，其中包括182个手动编写的测试方法和17个自动挖掘的测试方法。平均运行时开销较低。我们检查了652个违反手动编写的规范和（随机抽样）200个违反自动挖掘的规范。我们报告了95个bug，其中开发人员已经修复或接受了76个。然而，大多数违规行为，82.81%的652和97.89%的200，是假警报。基于我们的实证结果，我们得出结论：（1）运行时验证技术已经足够成熟，在测试过程中产生可容忍的运行时开销，（2）现有的API规范可以找到许多开发人员愿意修复的错误;然而，（3）误报率令人担忧，并建议大量的努力需要花费在工程更好的规范和正确评估其有效性。我们在一组不同的18个项目上重复了我们的实验，并检查了所有产生的742个违规行为。结果是相似的，我们的结论也是一样的。
Runtime verification can be used to find bugs early, during software development, by monitoring test executions against formal specifications (specs). The quality of runtime verification depends on the quality of the specs. While previous research has produced many specs for the Java API, manually or through automatic mining, there has been no large-scale study of their bug-findingeffectiveness. Our conference paper presented the first in-depth study of the bug-finding effectiveness of previously proposed specs. We used JavaMOP to monitor 182 manually written and 17 automatically mined specs against more than 18K manually written and 2.1M automatically generated test methods in 200 open-source projects. The average runtime overhead was under. We inspected 652 violations of manually written specs and (randomly sampled) 200 violations of automatically mined specs. We reported 95 bugs, out of which developers already fixed or accepted 76. However, most violations, 82.81% of 652 and 97.89% of 200, were false alarms. Based on our empirical results, we conclude that (1) runtime verification technology has matured enough to incur tolerable runtime overhead during testing, and (2) the existing API specifications can find many bugs that developers are willing to fix; however, (3) the false alarm rates are worrisome and suggest that substantial effort needs to be spent on engineering better specs and properly evaluating their effectiveness. We repeated our experiments on a different set of 18 projects and inspected all resulting 742 violations. The results are similar, and our conclusions are the same.