A Large-Scale Empirical Study of Real-Life Performance Issues in Open Source Projects
A Large-Scale Empirical Study of Real-Life Performance Issues in Open Source Projects
复制标题
DOI：
10.1109/tse.2022.3167628
复制
发表时间：
2023-02
影响因子：
7.4
通讯作者：
Yutong Zhao;Lu Xiao;A. Bondi;Bihuan Chen;Yang Liu
中科院分区：
文献类型：
--
作者：
Yutong Zhao;Lu Xiao;A. Bondi;Bihuan Chen;Yang Liu
软件性能是决定软件系统成功与否的关键质量属性。然而，实践者缺乏从技术、工程和经济的角度全面和全面地理解实际生活中的性能问题是如何引起和解决的。本文对来自不同问题领域的13个开源项目的570个实际性能问题进行了大规模的经验研究，并用三种流行的编程语言实现：Java(192个问题)、C/C++(162个问题)和Python语言(216个问题)。从技术角度来看，我们总结了八种常见的性能问题类型，以及相应的根本原因和适用于所有三种语言的解决方案。我们还从文献中确定了可用于检测和解决不同类型问题的工具。此外，我们发现，在570个问题中，有27%是通过设计级优化-对一组相关的源文件及其设计结构进行协调修订来解决的。我们揭示了四种典型的设计级优化模式，包括经典设计模式、更改传播、优化克隆和并行优化，实践者在解决性能问题时应该注意这些模式。本研究从工程角度分析了测试代码在性能优化方面的变化。我们发现，在570个性能问题中，只有15%涉及到测试代码的修订。在大多数情况下，修改后的测试用例侧重于性能优化的功能逻辑，而不是直接评估性能改进。这一发现表明，回归测试中可能缺乏正式验证性能优化的工程标准。最后，从经济学的角度对绩效优化的投资回报率进行了分析。我们发现，设计级优化通常需要更多投资，但并不总是带来更高的性能改进。然而，当开发人员关心其他质量属性时，例如可维护性和可读性，他们倾向于使用设计级优化。
Software performance is a critical quality attribute that determines the success of a software system. However, practitioners lack comprehensive and holistic understanding of how real-life performance issues are caused and resolved in practice from the technical, engineering, and economic perspectives. This paper presents a large-scale empirical study of 570 real-life performance issues from 13 open source projects from various problem domains, and implemented in three popular programming languages, Java (192 issues), C/C++ (162 issues), and Python (216 issues). From the technical perspective, we summarize eight general types of performance issues with corresponding root causes and resolutions that apply for all three languages. We also identify available tools for detecting and resolving different types of issues from the literature. In addition, we found that 27% of the 570 issues are resolved by design-level optimization—coordinated revision of a group of related source files and their design structure. We reveal four typical design-level optimization patterns, including classic design patterns, change propagation, optimization clone, and parallel optimization that practitioners should be aware of in resolving performance issues. From the engineering perspective, this study analyzes how test code changes in performance optimization. We found that only 15% of the 570 performance issues involve revision of test code. In most cases, the revised test cases focus on the functional logic of the performance optimization, rather than directly evaluate the performance improvement. This finding points to the potential lack of engineering standard for formally verifying performance optimization in regression testing. Finally, from the economic perspective, we analyze the “Return On Investment” of performance optimization. We found that design-level optimization usually requires more investment, but not always yields to higher performance improvement. However, developers tend to use design-level optimization when they concern about other quality attributes, such as maintainability and readability.