Analyzing causes of test failures

点击翻译按钮获取中文摘要

In this work, we propose to carry out a systematic analysis of test failures and suggest strategies to mitigate test flakiness. The work would investigate symptoms as observed in test execution logs, as well as potential causes including recent code changes; changes in tests' dependencies; and plugin load orders. It would also investigate correlations between test types and failure rates: for instance, some of Arista's tests are based on state machines. Do such tests fail more or less often than other tests? The immediate outcome of the work would be a diagnosis of the current state of failing tests along with suggestions to increase the reliability of the test suite. We also propose to build models of test cases which would encapsulate the reasons that they fail.