CRII: SHF: Toward Sustainable Software for Science - Implementing and Assessing Systematic Testing Approaches for Scientific Software

科学软件广泛应用于科学和工程领域。此外，从科学软件中获得的结果被用作研究出版物的证据。尽管这些软件的使用至关重要，但许多研究指出缺乏对科学软件的系统测试。因此，细微的程序错误可能不会被发现。有许多报道称，科学软件中的细微缺陷导致了数十亿美元的损失和科学出版物的撤回。本研究旨在开发自动化技术，用于测试oracle创建，测试用例选择，以及针对科学软件的测试oracle优先级的开发方法。本研究的学术价值如下：(1)它促进了对科学软件开发过程的理解，并研究了在不干扰科学探究的情况下将系统软件测试活动纳入科学软件开发的方法；(2)它为产生复杂输出的程序和产生未知输出的程序开发自动化测试预言机创造了新方法；(3)它开发了新的度量标准来度量部分测试oracle的有效性，并将它们用于测试oracle的优先级排序；(4)它扩展了当前测试用例选择的边界，以有效地使用部分或近似测试oracle。项目更广泛的意义和重要性是：(1)产生一个公开可用的，易于使用的测试工具，它可以被纳入科学的软件开发文化中，这样测试活动就不会干扰？科学,做什么?(2)招募美国原住民和女性从事计算机科学研究；(3)开设一门新的高等本科课程，名为？科学家的软件开发方法？针对非计算机科学学科的高年级本科生。这个项目开发了METtester：一个可以有效测试科学软件的自动化测试框架。这个测试框架分析被测程序的源代码，并利用机器学习技术来识别合适的被称为变质关系（MRs）的测试oracle。然后，它自动生成有效的测试用例，使用基于突变的方法基于识别的MRs进行自动化测试。之后，它创建用于测试的mr的优先顺序，以便在测试过程中尽早识别故障。最后，METtester使用MRs的优先顺序和生成的测试用例对被测程序进行测试。

Scientific software is widely used in science and engineering. Inaddition, results obtained from scientific software are used as evidence inresearch publications. Despite the critical usage of such software, manystudies have pointed out a lack of systematic testing of scientific software.As a result, subtle program errors can remain undetected. There are numerousreports of subtle faults in scientific software causing losses of billions ofdollars and the withdrawal of scientific publications. This research aims to developautomated techniques for test oracle creation, test case selection, and developmethods for test oracle prioritization targeting scientific software. The intellectualmerits of this research are the following: (1) It advances the understanding ofthe scientific software development process and investigates methods toincorporate systematic software testing activities into scientific softwaredevelopment without interfering with the scientific inquiry, (2) It forges newapproaches to develop automated test oracles for programs that produce complexoutputs and for programs that produce outputs that are previously unknown, (3)It develops new metrics to measure the effectiveness of partial test oraclesand uses them for test oracle prioritization, (4) It extends the boundaries ofcurrent test case selection to effectively work with partial or approximatetest oracles. The project's broader significance and importance are (1)produces a publicly available, easy to use testing tool that can be incorporatedinto the scientific software development culture such that the testingactivities will not interfere with ?doing science,? (2) recruits Native Americansand women into computer science research, (3) develops a new higher levelundergraduate course titled ?Software development methods for Scientists?targeting senior level undergraduate students in non-CS disciplines.This project develops METtester: an automated testing framework that can effectivelytest scientific software. This testing framework analyzes the source code ofthe program under test and utilizes machine learning techniques in order toidentify suitable test oracles called metamorphic relations (MRs). Then, it automaticallygenerates effective test cases to conduct automated testing based on theidentified MRs using a mutation based approach. After that, it creates aprioritized order of MRs to be used with testing in order to identify faults asearly as possible during the testing process. Finally, METtester conductstesting on the program under test using the prioritized order of MRs with the generatedtest cases.