VDB-MR: MapReduce-based distributed data integration using virtual database
VDB-MR: MapReduce-based distributed data integration using virtual database
复制标题
DOI：
10.1016/j.future.2010.04.001
复制
发表时间：
2010-10
期刊：
影响因子：
--
通讯作者：
Yulai Yuan;Yongwei Wu;Xiao-Qian Feng;Jing Li;Guangwen Yang;Weimin Zheng
中科院分区：
文献类型：
--
作者：
Yulai Yuan;Yongwei Wu;Xiao-Qian Feng;Jing Li;Guangwen Yang;Weimin Zheng
数据集成在许多商业应用和科学研究中变得非常重要。人们从不同的角度提出并开发了许多算法和系统来解决相关问题。虚拟数据库系统被公认为是数据集成的有效解决方案之一。现有的虚拟数据库系统中的执行模块效率很低。MapReduce（MR）是一种新的并行处理计算模型，在大规模数据执行上具有良好的性能。在本文中，我们提出了一个新的分布式数据集成系统，称为VDB-MR，这是基于MapReduce技术，有效地集成来自异构数据源的数据。使用VDB-MR，统一视图（即，单个虚拟数据库）可以提供给用户。我们还进行了一系列的实验来评估VDB-MR通过比较它与一个开源的数据集成系统OGSA-DAI和两个DBMS并行。实验结果表明，VDB-MR在并行性能上明显优于OGSA-DAI和DBMS。
Data Integration is becoming very important in many commercial applications and scientific research. A lot of algorithms and systems have been proposed and developed to address related issues from different aspects. Virtual database systems are well-recognized as one of the effective solutions of data integration. The existing execution modules in virtual database systems are very ineffective. MapReduce (MR) is a new computing model for parallel processing and has a good performance on large-scale data execution. In this paper, we propose a new distributed data integration system, called VDB-MR, which is based on the MapReduce technology, to efficiently integrate data from heterogeneous data sources. With VDB-MR, a unified view (i.e., a single virtual database) of multiple databases can be provided to users. We also conducted a series of experiments to evaluate VDB-MR by comparing it with an open source data integration system OGSA-DAI and two DBMSs in parallel. Experiment results show that VDB-MR significantly outperforms OGSA-DAI and the DBMSs in parallel.