Collaborative Research: A Comparative Study of Approaches to Cluster-Based Large Scale Data Analysis

协作研究：基于集群的大规模数据分析方法的比较研究

• 批准号: 0844013
• 负责人: Samuel Madden
• 金额: $ 15.12万
• 依托单位: Massachusetts Institute of Technology
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2009
• 资助国家: 美国
• 起止时间: 2009-02-01 至 2012-01-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0844013&HistoricalAwards=false
• 关键词: Collaborative; Research; Comparative; Study; Approaches

This goal of this research project is to understand the tradeoffs between the MapReduce and parallel DBMS approaches to performing large-scale data analysis over large clusters of computers, and to bring together ideas from both communities. Both MapReduce and parallel database systems provide scalable data processing over hundreds to thousands of nodes. Both provide a stylized, high-level programming environment that allows users to efficiently filter and combine datasets while masking much of the complexity of parallelizing computation over a cluster. But they differ in substantial ways as well, such as their approaches to dealing with fault tolerance, their data modeling requirements, their query flexibility, and their ability to function in a heterogeneous processing environment.This multi-university team of researchers is investigating the effect of these differences on the performance and scalability of these two approaches. The research team is running a set of experiments that compare an open source MapReduce implementation (Hadoop) to two commercial parallel database systems (DB2 and Vertica) on a benchmark that includes a range of tasks designed to assess the tradeoffs between both approaches. The research team is seeking to understand which differences between the two approaches to performing large scale data analysis are fundamental tradeoffs, and which differences are possible to combine inside a single solution, so that ideas from one community can benefit the other.

本研究项目的目标是了解MapReduce和并行DBMS方法之间的权衡，以在大型计算机集群上执行大规模数据分析，并将两个社区的想法结合在一起。MapReduce和并行数据库系统都提供了在数百到数千个节点上的可扩展数据处理。 两者都提供了一个程式化的高级编程环境，允许用户有效地过滤和联合收割机数据集，同时掩盖了在集群上并行计算的大部分复杂性。但它们在本质上也有不同，例如它们处理容错的方法，它们的数据建模要求，它们的查询灵活性，以及它们在异构处理环境中运行的能力。这个由多所大学组成的研究小组正在研究这些差异对这两种方法的性能和可扩展性的影响。该研究团队正在运行一组实验，将开源MapReduce实现（Hadoop）与两个商业并行数据库系统（DB2和Vertica）进行比较，其中包括一系列旨在评估两种方法之间权衡的任务。研究团队正在试图了解这两种执行大规模数据分析的方法之间的哪些差异是根本的权衡，以及哪些差异可能在单个解决方案中联合收割机组合，以便来自一个社区的想法可以使另一个社区受益。