Category partition black-box software testing: theory, tool support and experiments

点击翻译按钮获取中文摘要

Category partition (CP) is a black-box software testing technique based on equivalence class partitioning and boundary value analysis. It applies to many contexts: e.g., C unit testing, Java unit/integration testing, system testing of use cases. It has also been combined with other techniques such as state-based testing from extended finite state machines.***CP begins by identifying the parameters and environment variables of a functionality under test. Environment variables are factors in the environment of execution of the program under test that may impact its behaviour. These parameters and environment variables are then envisioned into categories, which are characteristics that are deemed important from a testing viewpoint. Each characteristic leads to the definition of choices (i.e., equivalent classes) splitting the domain of values (implicitly) defined by the characteristic. Constraints can then specify that some choices from two different categories should always be used together, can never be used together, or can only be used together under certain condition. The choices are then combined to form test frames according to a selection criterion, while satisfying constraints. A simple adequacy criterion requires that each choice appears at least once in the set of test frames. Each test frame is then associated with actual input values (for parameters and environment variables), according to the choice specifications, to produce test cases. ***When using CP, one needs to take many decisions that can impact the result (e.g., cost, fault detection) of the testing campaign. Obviously the selection of choices has an impact: e.g., poorly defined categories and choices will lead to faults that slip through the testing activity; Choosing an adequacy criterion matters because criteria are not all equally demanding (i.e., cost) or effective (i.e., fault detection); The technology to produce a set of test frames adequate for a criterion, while accounting for constraints, will have an impact; The procedure to identify input values for test frames may have an impact.***Unfortunately, very little is known about the actual, precise impact of those decisions on the result of applying CP.***The purpose of this project is to fill this gap. Tool support will be created to facilitate the use of CP and alternative decisions, on different case studies. Real and synthetic faults will be used to evaluate the impact of the many alternatives on the effectiveness of the testing technique at finding faults.***The results will be of high interest to the research community: it is the first time the many alternative applications of CP will be studied. Results will also be highly relevant to industry: tool support, experimental evidence leading to better decision making. Results will be highly interesting for teaching purposes: tailoring the teaching of the technique, availability of the tool, including an online version.