SoD-HCER: Evaluation of Complex Designs--A Comparative Study

任何科学的一个关键要求是测量的能力。没有测量，很难评估和改进。我们今天不仅测量和评估软件设计的能力很弱，而且对现有软件设计评估方法的优缺点缺乏全面的了解。为了更多地了解当前评估方法的缺点，PI将在本项目中研究两个来自系统领域的软件产品，它们共享可修改性和性能的重要设计标准，Click模块化路由器和Jetty http服务器和servlet容器。对于这两个方案，都有明确的设计目标和广泛的变更历史。使用变更历史数据，PI将在每个程序的早期版本中创建一个近似的可修改性和性能缺陷的“主列表”。例如，需要更改许多文件的特性请求可以标记为可修改性缺陷。然后，她将对每个主题程序的早期版本应用几种评估方法（例如，定量软件度量，基于体系结构的评估，以及半自动方法，如设计片段），并将每种方法的输出与设计缺陷的大致主列表进行比较。项目结果将包括一个基准和实验协议的建立，它可以用来评价使用变更历史日志评估软件设计的新技术或工具，以及增加对现有技术和工具评估软件设计的缺陷的理解。更广泛的影响：软件设计很重要，因为软件很重要。为了改进软件设计，我们必须有足够的工具来评估候选设计。这个项目将为软件设计评估提供新的工具和技术。使用这些新工具和技术的程序员将产生更好的软件设计，从而产生更好的软件。此外，这个实验产生的基准将证明对软件工程教育者是有价值的；教师将能够设计涉及基准的作业和项目，这将使学生接触到评估软件设计和经验软件工程研究中的问题。

A critical requirement for any science is the ability to measure. Without measurement, it is difficult to evaluate and improve. Not only is our ability to measure and evaluate software designs today weak, we also lack a thorough understanding of the shortcomings of the strengths and weaknesses of existing software design evaluation methods. To learn more about the shortcomings of current evaluation methods, the PI will in this project study two software products from the systems domain which share the important design criteria of modifiability and performance, the Click modular router and the Jetty http server and servlet container. For each of these two programs, there exist clear design goals and extensive change histories. Using change history data, the PI will create an approximate "master list" of modifiability and performance flaws in early versions of each program. For example, a feature request that required changes to many files could be flagged as a modifiability flaw. She will then apply several evaluation methods (e.g., quantitative software metrics, architecture-based evaluation, and semiautomatic methods such as design snippets) to early versions of each subject program, and compare the output of each method to the approximate master list of design flaws. Project outcomes will include establishment of a benchmark and experimental protocol that can be used to critique new techniques or tools for evaluating software designs using change history logs, and an increased understanding of the deficiencies of existing techniques and tools for evaluating software designs.Broader Impacts: Software designs matter because software matters. To improve software designs, we must have adequate tools for evaluating candidate designs. This project will provide insight into new tools and techniques for software design evaluation. Programmers using these new tools and techniques will produce better software designs and thus better software. In addition, the benchmark produced by this experiment will prove valuable to software engineering educators; instructors will be able to design assignments and projects that involve the benchmark, and which will expose students to issues in evaluating software designs and empirical software engineering research.