Test Effectiveness, Localization, Prioritization, and Risk in Ericssons's Complex Test Environment

爱立信为全球移动网络开发和提供设备、软件和服务。渥太华的爱立信公司开发的蜂窝基站是手机和物理骨干网之间的接口。这种高可靠性环境需要在昂贵的硬件上进行测试，这些硬件可以模拟手机与基站的交互。有些测试要运行几天，成本很高，而有些测试只需要几秒钟。当测试失败时，软件集成过程停止，工程师必须进行干预。我们的目标是将测试优先级、测试日志分析和经验软件工程结合起来，以挖掘历史工件，包括过去的代码更改、错误、同行评审和测试运行，从而创建统计模型(1)确定有效的环境有效的测试优先级，(2)区分环境和产品测试失败，(3)通过清除测试日志中的噪音来定位故障，(4)量化变更风险并识别需要新测试的软件单元。(5)将这些可操作的结果集成到我们提出的工具TestGuru中。对测试优先级工作的研究受到过去运行的信息以及与其他开发工件的链接的限制。爱立信收集大量日志和测试运行的历史记录，部分是为了提供软件的再现性和审计。研究团队在挖掘和连接软件工件（包括日志和测试）方面具有丰富的经验。结合爱立信丰富的数据存储，我们计划创建统计模型和分类器，以有效地安排测试和量化风险，并链接工件，以提供有关故障的根本原因和导致故障的因素的额外信息。通过Engage和MITACS，我们为测试人员提供异常测试的排名列表，他们每天早上使用这些列表来确定要调查的测试的优先级。通过挖掘日志，我们将能够帮助开发人员隔离当前失败的代码。通过挖掘版本历史，我们将能够识别可能需要更多测试的有风险的更改。这些成果将提高渥太华爱立信1000名员工的生产力，并推进最先进的测试。

Ericsson develops and supplies equipment, software, and services for global mobile networks. Ericsson in Ottawa develops cellular base stations that are the interface between cell phones and the physical backbone networks. This high reliability environment requires testing on expensive hardware that can simulate cell phones interacting with base stations. Some tests are expensive running for days, while others take seconds. When a test fails, the software integration process stops and an engineer must intervene. Our goal is combine test prioritization, test log analysis, and empirical software engineering to mine the historical artifacts including past code changes, bugs, peer reviews, and test runs to create statistical models (1) to determine effective environmentally valid test prioritizations, (2) to differentiate between environment and product test failures, (3) to locate faults by cleaning the noise out of test logs, (4) to quantify change risk and identify software units that need new tests, and (5) to integrate these actionable results into our proposed tool called TestGuru. Research into test prioritization work has been limited by information about past runs and links to other development artifacts. Ericsson collects massive histories of logs and test runs in part to provide for reproducibility of software and auditing. The research team has extensive experience in mining and linking software artifacts including logs and tests. Combined with Ericsson's rich data store, we plan to create statistical models and classifiers to efficiently order tests and quantify risk, and to link artifacts to provide additional information about the root cause of failures and the factors that contribute to failures. Through an Engage and MITACS, we provide testers with ranked lists of anomalous tests that they use each morning to prioritize the tests they will investigate. By mining the logs we will be able to help developers isolate the currently failing code. By mining version history we will be able to identify risky changes that may need more testing. These outcomes will increase the productivity of Ericsson Ottawa's 1000 employees and advance the state-of-the-art in testing.