SoD-HCER: Evaluation of Complex Designs--A Comparative Study

对任何科学来说，一个重要的要求就是测量的能力。没有衡量，就很难评价和改进。不仅我们今天衡量和评估软件设计的能力很弱，而且我们也缺乏对现有软件设计评估方法的优缺点的彻底了解。为了更多地了解当前评估方法的不足，PI将在这个项目中研究来自系统领域的两个软件产品，它们都具有可修改性和性能的重要设计标准，即Click模块化路由器和Jetty http服务器和Servlet容器。对于这两个项目，都有明确的设计目标和广泛的变更历史。使用变更历史数据，PI将创建每个程序早期版本中的可修改性和性能缺陷的大致“主列表”。例如，需要更改许多文件的功能请求可能会被标记为可修改性缺陷。然后，她将对每个主题程序的早期版本应用几种评估方法(例如，量化软件度量、基于体系结构的评估和半自动方法，如设计片段)，并将每种方法的输出与设计缺陷的大致主列表进行比较。项目成果将包括建立一个基准和实验协议，该协议可以用来批评使用变更历史日志评估软件设计的新技术或工具，以及加深对评估软件设计的现有技术和工具的缺陷的理解。广泛的影响：软件设计很重要，因为软件很重要。为了改进软件设计，我们必须有足够的工具来评估候选设计。该项目将为软件设计评估提供新的工具和技术。使用这些新工具和技术的程序员将产生更好的软件设计，从而产生更好的软件。此外，这个实验产生的基准将被证明对软件工程教育工作者有价值；教师将能够设计涉及基准的作业和项目，这将使学生接触到评估软件设计和经验性软件工程研究的问题。

A critical requirement for any science is the ability to measure. Without measurement, it is difficult to evaluate and improve. Not only is our ability to measure and evaluate software designs today weak, we also lack a thorough understanding of the shortcomings of the strengths and weaknesses of existing software design evaluation methods. To learn more about the shortcomings of current evaluation methods, the PI will in this project study two software products from the systems domain which share the important design criteria of modifiability and performance, the Click modular router and the Jetty http server and servlet container. For each of these two programs, there exist clear design goals and extensive change histories. Using change history data, the PI will create an approximate "master list" of modifiability and performance flaws in early versions of each program. For example, a feature request that required changes to many files could be flagged as a modifiability flaw. She will then apply several evaluation methods (e.g., quantitative software metrics, architecture-based evaluation, and semiautomatic methods such as design snippets) to early versions of each subject program, and compare the output of each method to the approximate master list of design flaws. Project outcomes will include establishment of a benchmark and experimental protocol that can be used to critique new techniques or tools for evaluating software designs using change history logs, and an increased understanding of the deficiencies of existing techniques and tools for evaluating software designs.Broader Impacts: Software designs matter because software matters. To improve software designs, we must have adequate tools for evaluating candidate designs. This project will provide insight into new tools and techniques for software design evaluation. Programmers using these new tools and techniques will produce better software designs and thus better software. In addition, the benchmark produced by this experiment will prove valuable to software engineering educators; instructors will be able to design assignments and projects that involve the benchmark, and which will expose students to issues in evaluating software designs and empirical software engineering research.