Analyzing causes of test failures

分析测试失败的原因

• 批准号: 485177-2015
• 负责人: Lam, Patrick
• 金额: $ 1.82万
• 依托单位: University of Waterloo
• 依托单位国家: 加拿大
• 项目类别: Engage Grants Program
• 财政年份: 2015
• 资助国家: 加拿大
• 起止时间: 2015-01-01 至 2016-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=584775
• 关键词: Analyzing; causes; test; failures

In this work, we propose to carry out a systematic analysis of test failures and suggest strategies to mitigate test flakiness. The work would investigate symptoms as observed in test execution logs, as well as potential causes including recent code changes; changes in tests' dependencies; and plugin load orders. It would also investigate correlations between test types and failure rates: for instance, some of Arista's tests are based on state machines. Do such tests fail more or less often than other tests? The immediate outcome of the work would be a diagnosis of the current state of failing tests along with suggestions to increase the reliability of the test suite. We also propose to build models of test cases which would encapsulate the reasons that they fail.

在这项工作中，我们建议对测试失败进行系统的分析，并提出减少测试剥落的策略。这项工作将调查在测试执行日志中观察到的症状，以及潜在的原因，包括最近的代码更改；测试依赖项的更改；以及插件加载顺序。它还将调查测试类型和失败率之间的相关性：例如，Arista的一些测试基于状态机。与其他测试相比，这类测试的失败频率是更高还是更低？ 这项工作的直接结果将是对失败测试的当前状态进行诊断，并提出增加测试套件可靠性的建议。我们还建议构建测试用例的模型，以封装测试用例失败的原因。