Test Effectiveness, Localization, Prioritization, and Risk in Ericssons's Complex Test Environment

点击翻译按钮获取中文摘要

Ericsson develops and supplies equipment, software, and services for global mobile networks. Ericsson in Ottawa develops cellular base stations that are the interface between cell phones and the physical backbone networks. This high reliability environment requires testing on expensive hardware that can simulate cell phones interacting with base stations. Some tests are expensive running for days, while others take seconds. When a test fails, the software integration process stops and an engineer must intervene. Our goal is combine test prioritization, test log analysis, and empirical software engineering to mine the historical artifacts including past code changes, bugs, peer reviews, and test runs to create statistical models (1) to determine effective environmentally valid test prioritizations, (2) to differentiate between environment and product test failures, (3) to locate faults by cleaning the noise out of test logs, (4) to quantify change risk and identify software units that need new tests, and (5) to integrate these actionable results into our proposed tool called TestGuru. Research into test prioritization work has been limited by information about past runs and links to other development artifacts. Ericsson collects massive histories of logs and test runs in part to provide for reproducibility of software and auditing. The research team has extensive experience in mining and linking software artifacts including logs and tests. Combined with Ericsson's rich data store, we plan to create statistical models and classifiers to efficiently order tests and quantify risk, and to link artifacts to provide additional information about the root cause of failures and the factors that contribute to failures. Through an Engage and MITACS, we provide testers with ranked lists of anomalous tests that they use each morning to prioritize the tests they will investigate. By mining the logs we will be able to help developers isolate the currently failing code. By mining version history we will be able to identify risky changes that may need more testing. These outcomes will increase the productivity of Ericsson Ottawa's 1000 employees and advance the state-of-the-art in testing.