Collaborative Research: A Comparative Study of Approaches to Cluster- Based Large Scale Data Analysis

该研究项目的目标是了解在大型计算机集群上执行大规模数据分析时，MapReduce和并行DBMS方法之间的权衡，并将这两个社区的想法结合在一起。MapReduceTM和并行数据库系统都在数百到数千个节点上提供可伸缩的数据处理。两者都提供了一个风格化的高级编程环境，允许用户高效地过滤和组合数据集，同时屏蔽了在集群上并行计算的大部分复杂性。但它们在处理容错的方法、数据建模需求、查询灵活性以及在异类处理环境中运行的能力等方面也存在实质性差异。这个由多所大学组成的研究团队正在调查这些差异对这两种方法的性能和可扩展性的影响。研究团队正在进行一系列实验，比较开放源码的MapReduce实现(Hadoop)和两个商业并行数据库系统(DB2和Vertica)，基准测试包括一系列旨在评估这两种方法之间的权衡的任务。研究团队正在寻求了解这两种执行大规模数据分析的方法之间的哪些差异是基本的权衡，以及哪些差异可能合并到一个解决方案中，以便一个社区的想法可以惠及另一个社区。

This goal of this research project is to understand the tradeoffs between the MapReduce and parallel DBMS approaches to performing large-scale data analysis over large clusters of computers, and to bring together ideas from both communities. Both MapReduce and parallel database systems provide scalable data processing over hundreds to thousands of nodes. Both provide a stylized, high-level programming environment that allows users to efficiently filter and combine datasets while masking much of the complexity of parallelizing computation over a cluster. But they differ in substantial ways as well, such as their approaches to dealing with fault tolerance, their data modeling requirements, their query flexibility, and their ability to function in a heterogeneous processing environment.This multi-university team of researchers is investigating the effect of these differences on the performance and scalability of these two approaches. The research team is running a set of experiments that compare an open source MapReduce implementation (Hadoop) to two commercial parallel database systems (DB2 and Vertica) on a benchmark that includes a range of tasks designed to assess the tradeoffs between both approaches. The research team is seeking to understand which differences between the two approaches to performing large scale data analysis are fundamental tradeoffs, and which differences are possible to combine inside a single solution, so that ideas from one community can benefit the other.