Better Science Through Benchmarking: Theory Validation and Application to Software Engineering

提案编号：0430026标题：更好的科学通过基准测试：理论验证和应用软件工程PI：苏珊埃利奥特西姆研究旨在完善一个初步的基准测试理论，并使用该理论来指导软件工程研究成果的基准测试的发展。在计算机科学的其他领域，基准已经被用来比较计算机系统、信息检索算法、数据库和许多其他技术的性能。在一个研究领域内，基准的创建和广泛使用往往伴随着快速的技术进步和社区建设。根据基准理论，这些效应的出现是因为基准与学科的科学范式密切相关（如托马斯库恩所描述的）。基准和范例都是通过对技术成果和文化行为期望的共识而产生的。基于一个成功的初步研究，将此理论应用于逆向工程，本研究开发了额外的基准在软件工程和收集数据来验证理论。这项工作将通过正在制定的具体基准、对研究结果的经验评估以及对研究如何进行的更好理解而为该领域带来好处。就更广泛的影响而言，这些基准使地域代表性不足的群体能够参与顶级研究，因为它们可以从网站下载并在合理的时间内完成。

Proposal Number: 0430026TITLE: Better Science Through Benchmarking: Theory Validation and Application to Software EngineeringPI: Susan Elliott SimThe research seeks to refine a preliminary theory of benchmarking and use the theory to guide development of benchmarks for research results in software engineering. Benchmarks have been used in elsewhere in computer science to compare the performance of computer systems, information retrieval algorithms, databases, and many other technologies. The creation and widespread use of a benchmark within a research area is frequently accompanied by rapid technical progress and community building. According to the theory of benchmarking, these effects appear because a benchmark is closely tied to the discipline's scientific paradigm (as described by Thomas Kuhn.) Both benchmarks and paradigms emerge through a consensus on technical results and cultural expectations for conduct. Based on a successful preliminary study applying this theory to reverse engineering, this research develops additional benchmarks in software engineering and collect data to validate the theory. This work will have benefits for the field through the specific benchmarks being developed, empirical evaluation of research results, and an improved understanding of how research is conducted. In terms of broader impact, the benchmarks enable geographically under-represented groups to participate in top tier research, because they can be downloaded from a web site and completed in reasonable time.