Collaborative Research: A Comparative Study of Approaches to Cluster-Based Large Scale Data Analysis

本研究项目的目标是了解MapReduce和并行DBMS方法之间的权衡，以便在大型计算机集群上执行大规模数据分析，并汇集两个社区的想法。MapReduce和并行数据库系统都在数百到数千个节点上提供可扩展的数据处理。两者都提供了一个程式化的高级编程环境，允许用户有效地过滤和组合数据集，同时屏蔽了在集群上并行计算的大部分复杂性。但是它们在本质上也存在差异，例如处理容错的方法、数据建模需求、查询灵活性以及在异构处理环境中工作的能力。这个由多所大学组成的研究团队正在研究这些差异对这两种方法的性能和可伸缩性的影响。研究团队正在运行一组实验，将开源MapReduce实现（Hadoop）与两个商业并行数据库系统（DB2和Vertica）在基准测试上进行比较，其中包括一系列旨在评估两种方法之间权衡的任务。研究团队正在寻求理解，在执行大规模数据分析的两种方法之间，哪些差异是根本的权衡，哪些差异可以在单一解决方案中结合起来，以便一个社区的想法可以使另一个社区受益。

This goal of this research project is to understand the tradeoffs between the MapReduce and parallel DBMS approaches to performing large-scale data analysis over large clusters of computers, and to bring together ideas from both communities. Both MapReduce and parallel database systems provide scalable data processing over hundreds to thousands of nodes. Both provide a stylized, high-level programming environment that allows users to efficiently filter and combine datasets while masking much of the complexity of parallelizing computation over a cluster. But they differ in substantial ways as well, such as their approaches to dealing with fault tolerance, their data modeling requirements, their query flexibility, and their ability to function in a heterogeneous processing environment.This multi-university team of researchers is investigating the effect of these differences on the performance and scalability of these two approaches. The research team is running a set of experiments that compare an open source MapReduce implementation (Hadoop) to two commercial parallel database systems (DB2 and Vertica) on a benchmark that includes a range of tasks designed to assess the tradeoffs between both approaches. The research team is seeking to understand which differences between the two approaches to performing large scale data analysis are fundamental tradeoffs, and which differences are possible to combine inside a single solution, so that ideas from one community can benefit the other.