Analyzing Tests for Correctness, Adequacy, and Effectiveness

软件开发人员不断地应用代码更改来添加新功能、改进产品或修复已知的错误。例如，据报道，50%的Google代码库每个月都会发生变化。为了确保代码库保持健康，编写测试用例以验证生产代码是否按预期运行。这样的测试也可以作为持续的回归测试，以确保当软件发展时，以前的工作功能仍然有效。 不充分的测试每年给美国经济造成600亿美元的损失。剑桥大学最近的一项研究发现，软件漏洞每年给全球经济造成3120亿美元的损失。该研究还发现，平均而言，软件开发人员花费50%的编程时间来查找和修复bug。 测试用例是为了形成第一道防线，防止软件产品中引入错误，特别是在重新测试修改后的代码时。因此，评估测试集的质量对于确保软件系统正确实现至关重要，这有助于降低总体开发和维护成本。 测试质量可以从不同的角度考虑。我们建议研究测试用例的正确性、充分性和有效性，这直接影响到整个测试集的质量。本研究计划的目标是：（1）研究与实际测试正确性、测试充分性和测试有效性相关的测试质量问题;（2）探索分析和评估测试用例质量的新技术;（3）提出新的解决方案，以直接提高测试用例的质量，并最终提高产品代码的测试质量。 虽然测试代码的质量与生产代码一样重要，但与生产代码不同，开发人员编写的测试的质量迄今为止尚未受到研究界的关注，特别是在测试用例的正确性方面。这项建议旨在深入研究和分析考试的质量。这项工作的结果将对软件测试人员和开发人员产生积极的影响。所提出的技术将使测试用例更加正确、充分和有效，从而提高产品代码的质量，并使最终用户更有信心地发布软件。

Software developers continuously apply code changes to add new features, improve the product, or fix known bugs. For instance, 50% of the Google code base is reported to change every month. In order to ensure that the code base remains healthy, test cases are written to verify that the production code functions as expected. Such tests also serve as continuous regression tests to ensure that previously working functionality still works, when the software evolves. Inadequate testing costs the US economy $60 billion annually. A more recent study by Cambridge University finds that software bugs cost the global economy $312 billion per year. The study also found that, on average, software developers spend 50% of their programming time finding and fixing bugs. Test cases are meant to form the first line of defence against the introduction of faults in a software product, especially when retesting modified code. Therefore, assessing the quality of a test suite is crucial to ensure that software systems are correctly implemented, which can help to decrease the overall development and maintenance cost. Test quality can be considered in different perspectives. We propose to study the correctness, adequacy, and effectiveness of test cases, which directly impact the overall test suite quality. The objectives of this research proposal are to (1) study test quality issues related to test correctness, test adequacy, and test effectiveness in practice, (2) explore novel techniques for analyzing and assessing the quality of test cases, (3) and propose new solutions for increasing the quality of test cases directly, and the production code being tested as a result. Although the quality of test code is as important as production code, unlike production code, the quality of tests written by developers has not received much attention from the research community thus far, especially when it comes to the correctness of test cases. This proposal seeks to study and analyze the quality of tests in depth. The results of this work will have positive implications for both software testers and developers. The techniques proposed will make test cases more correct, adequate, and effective and as a result the quality of the production code can be improved and the software can be released with much more confident to end-users.