Themis: an I/O-efficient MapReduce
Themis: an I/O-efficient MapReduce
复制标题
DOI：
10.1145/2391229.2391242
复制
发表时间：
2012-10
期刊：
影响因子：
--
通讯作者：
A. Rasmussen;V. Lam;Michael Conley;G. Porter;Rishi Kapoor;Amin Vahdat
中科院分区：
文献类型：
--
作者：
A. Rasmussen;V. Lam;Michael Conley;G. Porter;Rishi Kapoor;Amin Vahdat
“大数据”计算越来越多地利用MapReduce编程模型对大数据集合进行可扩展处理。许多MapReduce作业都是I/O绑定的，因此最小化I/O操作的数量对于提高它们的性能至关重要。在这项工作中，我们介绍了Themis，一个MapReduce实现，读取和写入数据记录到磁盘正好两次，这是无法在内存中容纳数据集的最小可能量。为了最小化I/O， Themis做出了与以前的MapReduce实现完全不同的设计决策。Themis执行各种各样的MapReduce任务，包括点击日志分析、DNA读取序列比对和PageRank，其速度接近TritonSort创纪录的排序性能[29]。
"Big Data" computing increasingly utilizes the MapReduce programming model for scalable processing of large data collections. Many MapReduce jobs are I/O-bound, and so minimizing the number of I/O operations is critical to improving their performance. In this work, we present Themis, a MapReduce implementation that reads and writes data records to disk exactly twice, which is the minimum amount possible for data sets that cannot fit in memory. In order to minimize I/O, Themis makes fundamentally different design decisions from previous MapReduce implementations. Themis performs a wide variety of MapReduce jobs -- including click log analysis, DNA read sequence alignment, and PageRank -- at nearly the speed of TritonSort's record-setting sort performance [29].