Automating Platform Selection for MapReduce Processing in the Cloud
Automating Platform Selection for MapReduce Processing in the Cloud
复制标题
DOI：
10.1109/iccac.2015.15
复制
发表时间：
2015-09
期刊：
影响因子：
--
通讯作者：
Zhuoyao Zhang;L. Cherkasova;B. T. Loo
中科院分区：
文献类型：
--
作者：
Zhuoyao Zhang;L. Cherkasova;B. T. Loo
云计算使用户能够快速配置任何所需大小的Hadoop集群，然后为使用这些资源的时间付费。在相同的预算下，用户可以租用更大量的资源并在更短的时间内处理其横向扩展应用程序，或者租用较小的集群但支付更长的处理时间。此外，云中存在各种不同类型的VM实例（例如，小型、中型或大型EC2实例）。所提供虚拟机的容量差异反映在虚拟机的定价中。因此，同样以相同的价格，用户可以获得基于不同VM实例类型的各种“类似容量”的Hadoop集群。我们观察到MapReduce应用程序的性能在不同的平台上可能会有很大的差异。这使得为给定的工作负载选择最佳的成本/性能平台成为一个重要的问题，特别是当它包含具有不同平台偏好的多个作业时。在这项工作中，我们设计了一个框架来解决以下问题：给定一组MapReduce作业的完成时间目标，确定同构或异构Hadoop集群配置（即，VM的数量、类型和作业调度），以便在给定的截止日期内处理这些作业，同时最小化租用的基础设施成本。我们概括建议的框架，考虑到可能的节点故障和性能下降的目标。我们对Amazon EC2平台进行的评估研究表明，对于不同的工作负载组合，在使用不同（但看似相同）的选择时，优化的平台选择可以节省45-68%的成本，以实现相同的性能目标。此外，根据工作负载的不同，异构解决方案的性能可能比同构集群解决方案高出26- 42%。我们分析并讨论了Amazon EC2平台上MapReduce处理的性能差异的可能原因。
Cloud computing enables a user to quickly provision any desirable size Hadoop cluster and then pay for the time these resources were used. With the same budget, a user can rent a larger amount of resources and process its scale-out application in a shorter time, or rent a smaller size cluster but pay a for longer processing time. Moreover, there is a variety of different types of VM instances in the Cloud (e.g., small, medium, or large EC2 instances). The capacity differences of the offered VMs are reflected in VM's pricing. Therefore, again for the same price a user can get a variety of "similar capacity" Hadoop clusters based on different VM instance types. We observe that performance of MapReduce applications may vary significantly on different platforms. This makes a selection of the best cost/performance platform for a given workload a non-trivial problem, especially when it contains multiple jobs with different platform preferences. In this work1, we design a framework for solving the following problem: given a completion time target for a set of MapReduce jobs, determine a homogeneous or heterogeneous Hadoop cluster configuration (i.e., the number, types of VMs, and the job schedule) for processing these jobs within a given deadline while minimizing the rented infrastructure cost. We generalize the proposed framework to take into account possible node failures and degraded performance goals. Our evaluation study with Amazon EC2 platform reveals that for different workload mixes, an optimized platform choice may result in 45-68% cost savings for achieving the same performance objectives when using different (but seemingly equivalent) choices. Moreover, depending on a workload the heterogeneous solution may outperform the homogeneous cluster solution by 26-42%. We analyze and discuss possible causes for observed performance differences of MapReduce processing on the Amazon EC2 platforms.