Precise Regression Benchmarking with Random Effects: Improving Mono Benchmark Results
Precise Regression Benchmarking with Random Effects: Improving Mono Benchmark Results
复制标题
具有随机效应的精确回归基准测试：改进单声道基准测试结果
DOI：
--
复制
发表时间：
2006
期刊：
影响因子：
--
通讯作者：
P. Tůma
中科院分区：
文献类型：
--
作者：
T. Kalibera;P. Tůma
众所周知，基准测试作为一种评估软件性能的方法会受到随机波动的影响，从而扭曲观察到的性能。在本文中，我们重点关注编译引起的波动。我们表明，如果实验期间观察到的性能要代表现实，基准测试的设计必须反映波动的存在
我们提出了一种新的基准实验统计模型，反映了编译、执行和测量中波动的存在。该模型描述了观察到的性能，并可以计算实验的最佳尺寸，从而在给定的时间内产生最佳的精度
使用各种基准，我们在回归基准的背景下评估模型。我们表明该模型显着减少了回归基准测试中错误检测到的性能变化的数量
Benchmarking as a method of assessing software performance is known to suffer from random fluctuations that distort the observed performance. In this paper, we focus on the fluctuations caused by compilation. We show that the design of a benchmarking experiment must reflect the existence of the fluctuations if the performance observed during the experiment is to be representative of reality
We present a new statistical model of a benchmark experiment that reflects the presence of the fluctuations in compilation, execution and measurement. The model describes the observed performance and makes it possible to calculate the optimum dimensions of the experiment that yield the best precision within a given amount of time
Using a variety of benchmarks, we evaluate the model within the context of regression benchmarking. We show that the model significantly decreases the number of erroneously detected performance changes in regression benchmarking