Taming Performance Variability
Taming Performance Variability
复制标题
DOI：
--
复制
发表时间：
2018-10
期刊：
影响因子：
--
通讯作者：
Aleksander Maricq;Dmitry Duplyakin;I. Jimenez;C. Maltzahn;Ryan Stutsman;R. Ricci;Ana Klimovic
中科院分区：
文献类型：
--
作者：
Aleksander Maricq;Dmitry Duplyakin;I. Jimenez;C. Maltzahn;Ryan Stutsman;R. Ricci;Ana Klimovic
计算硬件的性能各不相同：在同一台服务器（或具有假定相同部件的不同服务器）上重复运行的软件每次执行时可能产生不同的性能结果。这种差异对系统研究的可重复性以及定量比较不同系统性能的能力具有重要影响。它对商业计算也有影响，在商业计算中，协议通常是以达到特定性能目标为条件达成的。在10个月的时间里，我们进行了一项大规模研究，从835台服务器中获取了近90万个数据点。我们从两个角度研究这些数据：一个是希望提供一致环境的服务提供商的角度，另一个是必须了解可变性如何影响实验结果的系统研究人员的角度。通过这次研究，我们得出了一些关于性能可变性的类型和程度以及对实验结果可信度影响的经验教训。我们还创建了一个统计模型，可用于了解单个服务器在总体中的代表性如何。完整的数据集和我们的分析工具是公开可用的，并且我们已经构建了一个系统，可以交互式地探索数据，并根据对历史数据的统计分析为实验参数提供建议。
The performance of compute hardware varies: software run repeatedly on the same server (or a different server with supposedly identical parts) can produce performance results that differ with each execution. This variation has important effects on the reproducibility of systems research and ability to quantitatively compare the performance of different systems. It also has implications for commercial computing, where agreements are often made conditioned on meeting specific performance targets. Over a period of 10 months, we conducted a large-scale study capturing nearly 900,000 data points from 835 servers. We examine this data from two perspectives: that of a service provider wishing to offer a consistent environment, and that of a systems researcher who must understand how variability impacts experimental results. From this examination, we draw a number of lessons about the types and magnitudes of performance variability and the effects on confidence in experiment results. We also create a statistical model that can be used to understand how representative an individual server is of the general population. The full dataset and our analysis tools are publicly available, and we have built a system to interactively explore the data and make recommendations for experiment parameters based on statistical analysis of historical data.