CRII: SHF: Toward Sustainable Software for Science - Implementing and Assessing Systematic Testing Approaches for Scientific Software

科学软件在科学和工程领域有着广泛的应用。此外，从科学软件获得的结果被用作研究出版物中的证据。尽管这类软件的使用至关重要，但许多研究指出，缺乏对科学软件的系统测试。因此，细微的程序错误可能仍未被发现。有许多关于科学软件中的细微缺陷导致数十亿美元损失和科学出版物被撤回的报道。这项研究的目的是开发自动化技术，用于测试Oracle创建、测试用例选择，以及针对科学软件的测试Oracle优先级的开发方法。本研究的学术价值在于：(1)它促进了对科学软件开发过程的理解，研究了在不干扰科学研究的情况下将系统软件测试活动融入科学软件开发的方法；(2)它为产生复杂输出的程序和产生未知输出的程序开发自动测试预言提供了新的途径；(3)它开发了新的度量来衡量部分测试预言的有效性，并将它们用于测试预言的优先顺序；(4)它扩展了当前测试用例选择的边界，以有效地与部分或近似测试预言一起工作。该项目的更广泛的意义和重要性是(1)产生一个公开的，易于使用的测试工具，可以结合到科学软件开发文化中，使测试活动不会干扰做科学，(2)招募美洲原住民和妇女进入计算机科学研究，(3)开发一门新的更高水平的本科课程，名为？科学家的软件开发方法？面向非CS学科的高级本科生。该项目开发了METtester：一个可以有效测试科学软件的自动化测试框架。该测试框架分析被测程序的源代码，并利用机器学习技术来识别合适的测试预言，称为变形关系(MRS)。然后，使用基于突变的方法，根据识别出的MRS自动生成有效的测试用例进行自动化测试。然后，建立用于测试的MRS的优先顺序，以便在测试过程中尽早识别故障。最后，用生成的测试用例按照MRS的优先顺序对被测程序进行测试。

Scientific software is widely used in science and engineering. Inaddition, results obtained from scientific software are used as evidence inresearch publications. Despite the critical usage of such software, manystudies have pointed out a lack of systematic testing of scientific software.As a result, subtle program errors can remain undetected. There are numerousreports of subtle faults in scientific software causing losses of billions ofdollars and the withdrawal of scientific publications. This research aims to developautomated techniques for test oracle creation, test case selection, and developmethods for test oracle prioritization targeting scientific software. The intellectualmerits of this research are the following: (1) It advances the understanding ofthe scientific software development process and investigates methods toincorporate systematic software testing activities into scientific softwaredevelopment without interfering with the scientific inquiry, (2) It forges newapproaches to develop automated test oracles for programs that produce complexoutputs and for programs that produce outputs that are previously unknown, (3)It develops new metrics to measure the effectiveness of partial test oraclesand uses them for test oracle prioritization, (4) It extends the boundaries ofcurrent test case selection to effectively work with partial or approximatetest oracles. The project's broader significance and importance are (1)produces a publicly available, easy to use testing tool that can be incorporatedinto the scientific software development culture such that the testingactivities will not interfere with ?doing science,? (2) recruits Native Americansand women into computer science research, (3) develops a new higher levelundergraduate course titled ?Software development methods for Scientists?targeting senior level undergraduate students in non-CS disciplines.This project develops METtester: an automated testing framework that can effectivelytest scientific software. This testing framework analyzes the source code ofthe program under test and utilizes machine learning techniques in order toidentify suitable test oracles called metamorphic relations (MRs). Then, it automaticallygenerates effective test cases to conduct automated testing based on theidentified MRs using a mutation based approach. After that, it creates aprioritized order of MRs to be used with testing in order to identify faults asearly as possible during the testing process. Finally, METtester conductstesting on the program under test using the prioritized order of MRs with the generatedtest cases.