Mining Software Repositories and Information Visualization for Empirically Robust Testing of Variable Software

在现代软件系统中，“一刀切”是站不住脚的，因为它们必须进行定制，以适应不同和大量用户的需求。可变软件指的是这种类型的软件系统可以以多种方式进行配置，并且主要源于采用基于生成器的技术、成熟的软件产品线(SPL)方法、高级模块化范例、高度可配置的系统或特殊的重用实践，统称为克隆并拥有。测试可变软件特别具有挑战性，因为必须测试的配置数量通常很大，这一事实使得测试每个单独的配置是不可行的。 组合交互测试(CIT)被认为是解决这一挑战的一种范例，因为有多种可用的算法集，可以利用这些算法来计算用于测试的代表性配置组。然而，这项研究计划旨在解决几个悬而未决的问题。首先，严重缺乏公开可用的可变软件案例研究，以对CIT技术进行经验性评估和验证。其次，迫切需要对应用于可变软件的不同CIT技术进行彻底的实证和比较评估。第三，对可变软件的测试没有足够的工具支持。当前的做法是插件和独立工具的不协调和不完整分类的临时集合。 该研究计划的目标是： 1)利用挖掘软件库的技术来收集可公开获得的可变软件系统的案例研究，以对应用于可变软件的CIT技术进行彻底的经验评估。 2)对应用于可变软件的CIT技术进行系统评估。 3)利用信息可视化技术更好地传达各种软件案例研究中存在的大量信息。 4)提供强大的工具支持，允许软件工程师设计、实现和执行可变软件的测试。 第一个目标将通过研究GitHub等存储库中可用的开放源码项目以及开发挖掘配置和故障数据的工具来实现。对于第二个目标，我们将系统地收集和分类可变软件的CIT测试工具，并使用第一个目标的案例研究数据对它们进行评估。对于第三个目标，我们将探索和评估不同的可视化技术，以传达在测试可变软件时必须考虑的所有不同类型的信息，以做出适当的工程决策。对于第四个目标，我们将开发一个框架，允许灵活添加测试工具和可视化界面，为软件工程师提供强大的测试任务工具支持。

In modern software systems "one-size fits all" does not hold because they must be customized to fit the needs of diverse and large populations of users. Variable software refers to this type of software systems that can be configured in a large number of ways, and that stem primarily from the adoption of generator-based techniques, full-blown Software Product Line (SPL) approaches, advanced modularization paradigms, highly-configurable systems, or ad hoc reuse practices collectively called clone-and-own. Testing variable software is specially challenging because of the typically large number of configurations that must be tested, a fact that makes it unfeasible to test each individual configuration. Combinatorial Interaction Testing (CIT) has been advocated as a paradigm to address this challenge because of the diverse set of algorithms available which can be exploited to compute representative groups of configurations for testing. There are however several open problems that this research program aims to tackle. First, there is a critical lack of publicly-available case studies of variable software on which to empirically evaluate and validate CIT techniques. Second, there is a stark need of a thorough empirical and comparative evaluation of different CIT techniques that have been applied to variable software. Third, there is no adequate tool support for testing of variable software. The current practice is an ad-hoc collection of uncoordinated and incomplete assortment of plug-ins and stand-alone tools. The objectives for this research program are: 1) Exploit techniques for mining software repositories to gather publicly-available case studies of variable software systems for a thorough empirical evaluation of CIT techniques applied to variable software. 2) Perform a systematic assessment of CIT techniques applied to variable software. 3) Leverage information visualization techniques to better convey the large amount of information present in variable software case studies. 4) Provide robust tool support that allows software engineers to design, implement, and carry out test of variable software. The first objective will be achieved by studying open source projects available in repositories such as GitHub and developing tools that mine configuration and fault data. For the second objective we will systematically collect and catalog tools for CIT testing of variable software, and evaluate them with the case studies' data of the first objective. For the third objective we will explore and evaluate different visualization techniques to convey all the distinct types of information that must be considered for making adequate engineering decisions when testing variable software. For the fourth objective, we will develop a framework that allows flexible addition of testing tools and visualization interfaces to provide software engineers with robust tool support of testing tasks.