Energy efficiency for large-scale MapReduce workloads with significant interactive analysis
Energy efficiency for large-scale MapReduce workloads with significant interactive analysis
复制标题
DOI：
10.1145/2168836.2168842
复制
发表时间：
2012-04
期刊：
影响因子：
--
通讯作者：
Yanpei Chen;S. Alspaugh;Dhruba Borthakur;R. Katz
中科院分区：
文献类型：
--
作者：
Yanpei Chen;S. Alspaugh;Dhruba Borthakur;R. Katz
MapReduce工作负载已经发展到包括越来越多的时间敏感的交互式数据分析；我们将这种工作负载称为带有交互分析（MIA）的MapReduce。这种工作负载运行在大型集群上，其规模和成本使能源效率成为一个关键问题。之前关于MapReduce能源效率的工作还没有考虑到这类工作负载。提高硬件利用率有助于提高效率，但对于MIA工作负载来说，这是一个挑战。这些问题促使我们开发BEEMR (Berkeley Energy Efficient MapReduce)，这是一款节能的MapReduce工作负载管理器，其动机是对Facebook的真实MIA痕迹进行实证分析。关键的见解是，尽管MIA集群托管了大量数据，但交互式作业只操作一小部分数据，因此可以由一小部分专用机器提供服务；对时间不太敏感的作业可以以批处理方式在集群的其余部分上运行。BEEMR在严格的设计限制下实现了40-50%的节能，代表了为日益重要的数据中心工作负载提高能源效率的第一步。
MapReduce workloads have evolved to include increasing amounts of time-sensitive, interactive data analysis; we refer to such workloads as MapReduce with Interactive Analysis (MIA). Such workloads run on large clusters, whose size and cost make energy efficiency a critical concern. Prior works on MapReduce energy efficiency have not yet considered this workload class. Increasing hardware utilization helps improve efficiency, but is challenging to achieve for MIA workloads. These concerns lead us to develop BEEMR (Berkeley Energy Efficient MapReduce), an energy efficient MapReduce workload manager motivated by empirical analysis of real-life MIA traces at Facebook. The key insight is that although MIA clusters host huge data volumes, the interactive jobs operate on a small fraction of the data, and thus can be served by a small pool of dedicated machines; the less time-sensitive jobs can run on the rest of the cluster in a batch fashion. BEEMR achieves 40-50% energy savings under tight design constraints, and represents a first step towards improving energy efficiency for an increasingly important class of datacenter workloads.