Oracle problem in software testing
Oracle problem in software testing
复制标题
DOI：
10.1145/3092703.3098235
复制
发表时间：
2017-07
期刊：
影响因子：
--
通讯作者：
Gunel Jahangirova
中科院分区：
文献类型：
--
作者：
Gunel Jahangirova
预言机问题仍然是软件测试中的关键挑战之一，迄今为止几乎没有开发出自动化支持。在我的论文工作中，我们介绍了一种通过减少误报和漏报的发生率来评估和改进测试预言的技术。我们的技术结合了测试用例生成以揭示假阳性和突变测试以揭示假阴性。对五个真实世界受试者的实验结果表明，改进后预言机的故障检测率平均提高了 48.6%（比隐式预言机提高了 86%）。随后，开发人员确认并修复了所研究系统中的三个实际暴露的故障。然而，我们的技术包含一个人在循环中，在最初的实验中仅由作者代表。我们的下一个目标是进行进一步的实验，其中循环中的人将由真正的开发人员代表。我们未来的第二个目标是解决预言机放置问题。测试软件时，开发人员可以将预言机放置在方法的外部或内部。给定一个错误的执行状态，即与预期不同的执行状态，如果预言机被放置在无法访问错误程序状态或程序状态不再损坏的程序点，则它可能无法暴露错误。在这种情况下，预言机会遭受失败的错误传播。原则上，内部预言机比外部预言机更少受到失败的错误传播的影响。然而，它们也更难以手动定义。因此，一个关键的研究问题是，更具侵入性的预言机放置是否因其更高的故障检测能力而合理。
The oracle problem remains one of the key challenges in software testing, for which little automated support has been developed so far. In my thesis work we introduce a technique for assessing and improving test oracles by reducing the incidence of both false positives and false negatives. Our technique combines test case generation to reveal false positives and mutation testing to reveal false negatives. The experimental results on five real-world subjects show that the fault detection rate of the oracles after improvement increases, on average, by 48.6% (86% over the implicit oracle). Three actual, exposed faults in the studied systems were subsequently confirmed and fixed by the developers. However, our technique contains a human in the loop, which was represented only by the author during the initial experiments. Our next goal is to conduct further experiments where the human in the loop will be represented by real developers. Our second future goal is to address the oracle placement problem. When testing software, developers can place oracles externally or internally to a method. Given a faulty execution state, i.e., one that differs from the expected one, an oracle might be unable to expose the fault if it is placed at a program point with no access to the incorrect program state or where the program state is no longer corrupted. In such a case, the oracle is subject to failed error propagation. Internal oracles are in principle less subject to failed error propagation than external oracles. However, they are also more difficult to define manually. Hence, a key research question is whether a more intrusive oracle placement is justified by its higher fault detection capability.