SHF: MEDIUM: Achieving Software Reliability without True Test Oracles

传统的软件测试检查每个输出是否对一组测试输入正确。但对于某些软件，不知道某些输入的正确输出应该是什么——然而，检测该软件中的编码错误仍然很重要，这样它们就可以修复。这种困境经常出现在机器学习、模拟和优化应用程序中，通常是“为了首先确定答案而编写的程序”。如果正确答案是已知的，就没有必要编写这样的程序了。”由于这些类型的应用程序经常用于公共基础设施和生物医学研究（本研究的目标领域），因此在灾难发生之前检测和修复错误至关重要。幸运的是，许多这样的应用反映了“变形属性”，定义了输入和输出对之间的关系，这样对于任何已知输出0的先前输入i，可以很容易地推导出测试输入i‘并预测预期输出o’。如果实际输出0 ‘’与0 ‘’不同，则代码中一定存在错误。这个项目研究了确定软件的变形特性和设计好的测试用例的方法，从这些用例中可以推导出二次测试。该项目扩展了之前关于变形测试的工作中考虑的输入/输出，将重点放在应用程序状态之前和之后，而不仅仅是功能参数和结果。该研究还将由变形属性隐含的两两关系扩展到非确定性应用程序的“语义相似性”，应用于来自多个执行的配置文件，因为不能期望对单个测试执行对保持精确的关系。这些扩展允许对更复杂的特性进行处理，这些特性已经被初步的实验显示出来，从而揭示了在其他情况下无法检测到的缺陷。

Conventional software testing checks whether each output is correct for the set of test inputs. But for some software, it is not known what the correct output should be for some inputs -- yet it is still important to detect coding errors in that software, so they can be fixed. This dilemma arises frequently for machine learning, simulation and optimization applications, often "Programs which were written in order to determine the answer in the first place. There would be no need to write such programs, if the correct answer were known." As these kinds of applications are frequently used in public infrastructure and biomedical research (domains targeted in this research), it is critical to detect and fix errors before a calamity occurs. Fortunately, many such applications reflect 'metamorphic properties' that define a relationship between pairs of inputs and outputs, such that for any previous input i with its already known output o, one can easily derive a test input i' and predict the expected output o'. If the actual output o'' is different from o', then there must be an error in the code. This project investigates methodology for determining the metamorphic properties of software and for devising good test cases from which the secondary tests can be derived. The project extends the inputs/outputs considered in previous work on metamorphic testing to focus on application state, before and after, rather than just functional parameters and results. The research also extends the pairwise relations implied by metamorphic properties to 'semantic similarity' for nondeterministic applications, applied to profiles from numerous executions, since an exact relation cannot be expected to hold for a single pair of test executions. These extensions enable treatment of more sophisticated properties that preliminary experiments have shown to reveal defects that were not detected otherwise.