FlumeJava: Easy, Efficient Data-Parallel Pipelines
FlumeJava: Easy, Efficient Data-Parallel Pipelines
复制标题
DOI：
10.1145/1809028.1806638
复制
发表时间：
2010-06-01
影响因子：
--
通讯作者：
Weizenbaum, Nathan
中科院分区：
文献类型：
--
作者：
Chambers, Craig;Raniwala, Ashish;Weizenbaum, Nathan
MapReduce和类似的系统极大地简化了编写数据并行代码的任务。然而，许多真实世界的计算需要一条MapReduce管道，编程和管理这类管道可能会很困难。我们介绍了FlumeJava，这是一个Java库，它使开发、测试和运行高效的数据并行管道变得容易。FlumeJava库的核心是两个类，它们表示不变的并行集合，每个类都支持少量的操作来并行处理它们。并行集合及其操作提供了对不同数据表示和执行策略的简单、高级、统一的抽象。为了使并行操作能够高效运行，FlumeJava推迟了它们的评估，而是在内部构建了一个执行计划数据流图。当最终需要并行操作的最终结果时，FlumeJava首先优化执行计划，然后在适当的底层原语(如MapReduces)上执行优化的操作。并行数据和计算的高级抽象、延迟计算和优化以及高效的并行原语的组合产生了一个易于使用的系统，其效率接近手动优化的流水线。FlumeJava在Google内部被数百名管道开发人员活跃使用。
MapReduce and similar systems significantly ease the task of writing data-parallel code. However, many real-world computations require a pipeline of MapReduces, and programming and managing such pipelines can be difficult. We present FlumeJava, a Java library that makes it easy to develop, test, and run efficient data-parallel pipelines. At the core of the FlumeJava library are a couple of classes that represent immutable parallel collections, each supporting a modest number of operations for processing them in parallel. Parallel collections and their operations present a simple, high-level, uniform abstraction over different data representations and execution strategies. To enable parallel operations to run efficiently, FlumeJava defers their evaluation, instead internally constructing an execution plan dataflow graph. When the final results of the parallel operations are eventually needed, FlumeJava first optimizes the execution plan, and then executes the optimized operations on appropriate underlying primitives (e. g., MapReduces). The combination of high-level abstractions for parallel data and computation, deferred evaluation and optimization, and efficient parallel primitives yields an easy-to-use system that approaches the efficiency of hand-optimized pipelines. FlumeJava is in active use by hundreds of pipeline developers within Google.