CRII: SHF: Toward Sustainable Software for Science - Implementing and Assessing Systematic Testing Approaches for Scientific Software

科学软件广泛应用于科学和工程领域。此外，从科学软件中获得的结果被用作研究出版物的证据。尽管这类软件的使用至关重要，但许多研究指出，科学软件缺乏系统的测试。结果，细微的程序错误可能无法被发现。有许多报告称，科学软件中的细微缺陷导致数十亿美元的损失和科学出版物的撤回。本研究旨在发展自动化的测试预言产生技术，测试用例选择，以及针对科学软件的测试预言优先级的开发方法。这项研究的智力价值如下：（1）它促进了对科学软件开发过程的理解，并研究了在不干扰科学研究的情况下将系统的软件测试活动纳入科学软件开发的方法，（2）它为产生复杂输出的程序和产生以前未知输出的程序开发自动化测试预言机创造了新的方法，（3）提出了新的度量部分测试预言机有效性的指标，并将其用于测试预言机的优先级排序。（4）扩展了现有测试用例选择的边界，使其能够有效地处理部分或近似测试预言机。该项目的更广泛的意义和重要性是（1）产生一个公开可用的，易于使用的测试工具，可以融入科学的软件开发文化，这样的测试活动将不会干扰？做科学研究(2)招募美国原住民和妇女进入计算机科学研究，（3）开发一个新的更高层次的本科课程，名为？科学家的软件开发方法针对非计算机科学专业的高年级本科生，本项目开发了一个可以有效测试科学软件的自动化测试框架METtester。该测试框架分析了被测程序的源代码，并利用机器学习技术来识别合适的测试神谕，称为变形关系（MR）。然后，它自动生成有效的测试用例进行自动化测试的基础上确定的MR使用基于变异的方法。在此基础上，建立了用于测试的MR优先级排序，以便在测试过程中尽早识别故障。最后，METtester利用生成的测试用例，按照MR的优先顺序对被测程序进行测试。

Scientific software is widely used in science and engineering. Inaddition, results obtained from scientific software are used as evidence inresearch publications. Despite the critical usage of such software, manystudies have pointed out a lack of systematic testing of scientific software.As a result, subtle program errors can remain undetected. There are numerousreports of subtle faults in scientific software causing losses of billions ofdollars and the withdrawal of scientific publications. This research aims to developautomated techniques for test oracle creation, test case selection, and developmethods for test oracle prioritization targeting scientific software. The intellectualmerits of this research are the following: (1) It advances the understanding ofthe scientific software development process and investigates methods toincorporate systematic software testing activities into scientific softwaredevelopment without interfering with the scientific inquiry, (2) It forges newapproaches to develop automated test oracles for programs that produce complexoutputs and for programs that produce outputs that are previously unknown, (3)It develops new metrics to measure the effectiveness of partial test oraclesand uses them for test oracle prioritization, (4) It extends the boundaries ofcurrent test case selection to effectively work with partial or approximatetest oracles. The project's broader significance and importance are (1)produces a publicly available, easy to use testing tool that can be incorporatedinto the scientific software development culture such that the testingactivities will not interfere with ?doing science,? (2) recruits Native Americansand women into computer science research, (3) develops a new higher levelundergraduate course titled ?Software development methods for Scientists?targeting senior level undergraduate students in non-CS disciplines.This project develops METtester: an automated testing framework that can effectivelytest scientific software. This testing framework analyzes the source code ofthe program under test and utilizes machine learning techniques in order toidentify suitable test oracles called metamorphic relations (MRs). Then, it automaticallygenerates effective test cases to conduct automated testing based on theidentified MRs using a mutation based approach. After that, it creates aprioritized order of MRs to be used with testing in order to identify faults asearly as possible during the testing process. Finally, METtester conductstesting on the program under test using the prioritized order of MRs with the generatedtest cases.