HPMR: Prefetching and pre-shuffling in shared MapReduce computation environment
HPMR: Prefetching and pre-shuffling in shared MapReduce computation environment
复制标题
DOI：
10.1109/clustr.2009.5289171
复制
发表时间：
2009-10
期刊：
影响因子：
--
通讯作者：
Sangwon Seo;Ingook Jang;Kyungchang Woo;Inkyo Kim;Jin-Soo Kim;S. Maeng
中科院分区：
文献类型：
--
作者：
Sangwon Seo;Ingook Jang;Kyungchang Woo;Inkyo Kim;Jin-Soo Kim;S. Maeng
MapReduce是一种编程模型，支持机器学习、数据挖掘和科学模拟等大规模数据密集型应用的分布式并行处理。 Hadoop 是 MapReduce 编程模型的开源实现。包括 Yahoo!、Amazon 和 Facebook 在内的许多公司都使用 Hadoop 对用户搜索日志和访问日志等大规模数据集进行各种数据挖掘。在这些情况下，出于对成本、系统利用率和可管理性的实际考虑，多个用户共享相同的计算资源是很常见的。然而，Hadoop 假设所有集群节点都专用于单个用户，无法保证共享 MapReduce 计算环境中的高性能。在本文中，我们提出了预取和预洗牌两种优化方案，提高了共享环境下的整体性能，同时保留了与原生Hadoop的兼容性。所提出的方案在原生 Hadoop-0.18.3 中作为名为 HPMR（高性能 MapReduce 引擎）的插件组件实现。我们使用来自 Yahoo! 的三种不同工作负载和七种类型的测试集对 Yahoo!Grid 平台进行评估。表明 HPMR 将执行时间减少了高达 73%。
MapReduce is a programming model that supports distributed and parallel processing for large-scale data-intensive applications such as machine learning, data mining, and scientific simulation. Hadoop is an open-source implementation of the MapReduce programming model. Hadoop is used by many companies including Yahoo!, Amazon, and Facebook to perform various data mining on large-scale data sets such as user search logs and visit logs. In these cases, it is very common to share the same computing resources by multiple users due to practical considerations about cost, system utilization, and manageability. However, Hadoop assumes that all cluster nodes are dedicated to a single user, failing to guarantee high performance in the shared MapReduce computation environment. In this paper, we propose two optimization schemes, prefetching and pre-shuffling, which improve the overall performance under the shared environment while retaining compatibility with the native Hadoop. The proposed schemes are implemented in the native Hadoop-0.18.3 as a plug-in component called HPMR (High Performance MapReduce Engine). Our evaluation on the Yahoo!Grid platform with three different workloads and seven types of test sets from Yahoo! shows that HPMR reduces the execution time by up to 73%.