Nephele/PACTs: a programming model and execution framework for web-scale analytical processing
Nephele/PACTs: a programming model and execution framework for web-scale analytical processing
复制标题
DOI：
10.1145/1807128.1807148
复制
发表时间：
2010-06
期刊：
影响因子：
--
通讯作者：
Dominic Battré;Stephan Ewen;Fabian Hueske;O. Kao;V. Markl;Daniel Warneke
中科院分区：
文献类型：
--
作者：
Dominic Battré;Stephan Ewen;Fabian Hueske;O. Kao;V. Markl;Daniel Warneke
我们提出了一个并行数据处理器，其中心是所谓的并行化合同（PACTs）和可扩展并行执行引擎Nephele的编程模型[18]。PACT编程模型是著名的map/reduce编程模型的推广，它扩展了更多的二阶函数，以及输出合同，保证了函数的行为。我们描述的方法将PACT程序转换成一个数据流的Nephele，并行执行其顺序构建块，并处理通信，同步和容错。我们对PACTs的定义允许在转换期间对数据流应用几种类型的优化。该系统作为一个整体被设计为与map/reduce系统一样通用（并兼容），同时克服了它们的几个主要弱点：1）单独的map和reduce函数不足以自然有效地表达许多数据处理任务。2)Map/Reduce将程序绑定到一个单一的固定执行策略，这是健壮的，但对于许多任务来说是非常次优的。3)Map/Reduce对函数的行为没有任何假设。因此，它只提供非常有限的优化机会。通过一组示例和实验，我们说明了我们的系统是如何能够自然地表示和有效地执行几个任务，不适合地图/减少模型。
We present a parallel data processor centered around a programming model of so called Parallelization Contracts (PACTs) and the scalable parallel execution engine Nephele [18]. The PACT programming model is a generalization of the well-known map/reduce programming model, extending it with further second-order functions, as well as with Output Contracts that give guarantees about the behavior of a function. We describe methods to transform a PACT program into a data flow for Nephele, which executes its sequential building blocks in parallel and deals with communication, synchronization and fault tolerance. Our definition of PACTs allows to apply several types of optimizations on the data flow during the transformation. The system as a whole is designed to be as generic as (and compatible to) map/reduce systems, while overcoming several of their major weaknesses: 1) The functions map and reduce alone are not sufficient to express many data processing tasks both naturally and efficiently. 2) Map/reduce ties a program to a single fixed execution strategy, which is robust but highly suboptimal for many tasks. 3) Map/reduce makes no assumptions about the behavior of the functions. Hence, it offers only very limited optimization opportunities. With a set of examples and experiments, we illustrate how our system is able to naturally represent and efficiently execute several tasks that do not fit the map/reduce model well.