Conducting Repeatable Experiments in Highly Variable Cloud Computing Environments
Conducting Repeatable Experiments in Highly Variable Cloud Computing Environments
复制标题
在高度可变的云计算环境中进行可重复的实验
DOI：
10.1145/3030207.3030229
复制
发表时间：
2017
期刊：
影响因子：
--
通讯作者：
Tim Brecht
中科院分区：
文献类型：
--
作者：
A. Abedi;Tim Brecht
先前的工作表明，公共云计算环境中的基准测试和应用性能可能差异很大。利用包含受CPU、内存、磁盘和网络性能影响的测量数据的亚马逊EC2跟踪记录，我们研究了在云计算环境中比较性能测量的常用方法。结果显示这些方法存在相当大的缺陷，可能会导致错误的结论。例如，这些方法错误地报告说，在95%的置信水平下，两个相同系统的性能差异达38%。然后，我们使用相同的跟踪记录研究了随机多重交错试验（RMIT）方法的有效性。我们证明，尽管用户无法控制的变化条件使得比较竞争方案极具挑战性，但RMIT可用于在这种云计算环境中进行可重复的实验，从而实现公平比较。
Previous work has shown that benchmark and application performance in public cloud computing environments can be highly variable. Utilizing Amazon EC2 traces that include measurements affected by CPU, memory, disk, and network performance, we study commonly used methodologies for comparing performance measurements in cloud computing environments. The results show considerable flaws in these methodologies that may lead to incorrect conclusions. For instance, these methodologies falsely report that the performance of two identical systems differ by 38% using a confidence level of 95%. We then study the efficacy of the Randomized Multiple Interleaved Trials (RMIT) methodology using the same traces. We demonstrate that RMIT could be used to conduct repeatable experiments that enable fair comparisons in this cloud computing environment despite the fact that changing conditions beyond the user's control make comparing competing alternatives highly challenging.