IaaS cloud benchmarking: approaches, challenges, and experience
IaaS cloud benchmarking: approaches, challenges, and experience
复制标题
DOI：
10.1145/2462307.2462309
复制
发表时间：
2013-04
期刊：
影响因子：
--
通讯作者：
A. Iosup
中科院分区：
文献类型：
--
作者：
A. Iosup
在过去五年中，基础设施即服务 (IaaS) 云已发展成为 ICT 的一个分支，提供与按需租赁存储、计算和网络相关的服务。选择甚至使用（商业）IaaS 云的主要障碍之一是缺乏基准测试结果，即缺乏允许（潜在）云用户对 IaaS 云进行比较和推理的可信定量信息。在本次演讲中，我们讨论了对 IaaS 云进行定量评估的实证方法，以进行云基准测试。工业界和学术界多年来一直在使用经验方法，但由于挑战的复杂性和规模，IaaS 云和类似系统（例如网格）的成功有限。我们展示了我们对云工作负载特征（包括大数据应用程序）的研究的初步结果。我们介绍了在开发用于云性能评估的 SkyMark 框架中吸取的经验教训，以及我们基于 SkyMark 对三个研究问题进行调查的结果：生产 IaaS 云服务的性能如何？广泛使用的生产云服务的性能变化有多大？与 IaaS 云交互的配置和分配策略对性能有何影响？与之前的尝试相比，我们的研究结合了经验方法和其他方法，例如建模和模拟，以获得更多见解；基于短期和多年测量的结合，以确保我们的结果的长期性；并对多个真实云进行大规模、全面的研究，以减少与实验环境相关的有效性威胁。本演示还可以为与基准测试相关的领域提供有用的见解，例如在其他大型分布式系统中进行的实验评估。最后但并非最不重要的一点是，我们提出了云基准测试的路线图，以及我们计划与标准性能评估公司 (SPEC) 的 RG 云工作组的其他成员一起推进该路线图 (http://research.spec.org/working-groups/rg-cloud-working-group.html)。注：根据同音词文章[1]谈论。
Over the past five years, Infrastructure-as-a-Service (IaaS) clouds have grown into the branch of ICT that offers services related to on-demand lease of storage, computation, and network. One of the major impediments in the selection and even use of (commercial) IaaS clouds is the lack of benchmarking results, that is, the lack of trustworthy quantitative information that allows (potential) cloud users to compare and reason about IaaS clouds. In this talk we discuss empirical approaches to quantitative evaluation of IaaS clouds, toward cloud benchmarking. Both industry and academia have used empirical approaches for years, but with limited success for IaaS clouds and similar systems (e.g., grids) due to the complexity and size of challenges. We present initial results of our research into cloud workload characterization, including Big Data applications. We present the lessons we have learned in developing the SkyMark framework for cloud performance evaluation and the results of our SkyMark-based investigation of three research questions: What is the performance of production IaaS cloud services? How variable is the performance of widely used production cloud services? and What is the impact on performance of the provisioning and allocation policies that interact with IaaS clouds? In contrast to previous attempts, our research combines empirical and other approaches, for example modeling and simulation, for gaining more insight; is based on a combination of short-term and multi-year measurements, for ensuring the longevity of our results; and uses large, comprehensive studies of several real clouds, for reducing the threats to validity related to the experimental environment. This presentation can also provide useful insights for fields related to benchmarking, for example experimental evaluation conducted in other large-scale distributed systems. Last but not least, we present a road-map toward cloud benchmarking and the way we plan to progress on it with other members of the RG Cloud WG of the Standard Performance Evaluation Corporation (SPEC) http://research.spec.org/working-groups/rg-cloud-working-group.html). Note: talk based on the homonym article [1].