HybridMR: A Hierarchical MapReduce Scheduler for Hybrid Data Centers
HybridMR: A Hierarchical MapReduce Scheduler for Hybrid Data Centers
复制标题
DOI：
10.1109/icdcs.2013.31
复制
发表时间：
2013-07
期刊：
影响因子：
--
通讯作者：
Bikash Sharma;Timothy Wood;C. Das
中科院分区：
文献类型：
--
作者：
Bikash Sharma;Timothy Wood;C. Das
虚拟化环境之所以有吸引力，是因为它们简化了集群管理，同时促进了经济高效的工作负载整合。因此，公共云或私有数据中心中的虚拟机已成为运行 Web 服务和虚拟桌面等事务性应用程序的标准。另一方面，像 MapReduce 这样的批处理工作负载通常部署在本机集群中，以避免虚拟化的性能开销。虽然这些虚拟和本机环境都有自己的优点和缺点，但我们在这项工作中证明，在混合平台中提供这两种计算范例的最佳方案是可行的。在本文中，我们以由本机和虚拟环境组成的混合数据中心为例，并提出了一种称为 HybridMR 的两阶段分层调度程序，用于交互式和批处理工作负载的有效资源管理。在第一阶段，HybridMR 根据预期的虚拟化开销对传入的 MapReduce 作业进行分类，并使用此信息自动指导物理机和虚拟机之间的放置。在第二阶段，HybridMR 管理与交互式应用程序并置的 MapReduce 作业的运行时性能，以便尽力交付批处理作业，同时遵守交互式应用程序的服务级别协议 (SLA)。通过将批处理作业与超额配置的前台应用程序整合，可以更好地利用可用的未使用资源，从而提高应用程序性能和能源效率。对由 24 台物理服务器和 48 台虚拟机组成的混合集群以及交互式和批处理 MapReduce 应用程序的不同工作负载组合的评估表明，与纯虚拟情况相比，HybridMR 可以将 MapReduce 作业的完成时间提高高达 40%，同时符合交互式应用程序的 SLA。与纯原生集群相比，HybridMR 以最小的性能损失为代价，将资源利用率提高了 45%，并实现了高达 43% 的节能。这些结果表明，具有高效调度机制的混合数据中心可以为托管批处理和交互式工作负载提供经济高效的解决方案。
Virtualized environments are attractive because they simplify cluster management, while facilitating cost-effective workload consolidation. As a result, virtual machines in public clouds or private data centers, have become the norm for running transactional applications like web services and virtual desktops. On the other hand, batch workloads like MapReduce, are typically deployed in a native cluster to avoid the performance overheads of virtualization. While both these virtual and native environments have their own strengths and weaknesses, we demonstrate in this work that it is feasible to provide the best of these two computing paradigms in a hybrid platform. In this paper, we make a case for a hybrid data center consisting of native and virtual environments, and propose a 2-phase hierarchical scheduler, called HybridMR, for the effective resource management of interactive and batch workloads. In the first phase, HybridMR classifies incoming MapReduce jobs based on the expected virtualization overheads, and uses this information to automatically guide placement between physical and virtual machines. In the second phase, HybridMR manages the run-time performance of MapReduce jobs collocated with interactive applications in order to provide best effort delivery to batch jobs, while complying with the Service Level Agreements (SLAs) of interactive applications. By consolidating batch jobs with over-provisioned foreground applications, the available unused resources are better utilized, resulting in improved application performance and energy efficiency. Evaluations on a hybrid cluster consisting of 24 physical servers and 48 virtual machines, with diverse workload mix of interactive and batch MapReduce applications, demonstrate that HybridMR can achieve up to 40% improvement in the completion times of MapReduce jobs, over the virtual-only case, while complying with the SLAs of interactive applications. Compared to the native-only cluster, at the cost of minimal performance penalty, HybridMR boosts resource utilization by 45%, and achieves up to 43% energy savings. These results indicate that a hybrid data center with an efficient scheduling mechanism can provide a cost-effective solution for hosting both batch and interactive workloads.