Block-based load balancing for entity resolution with MapReduce
Block-based load balancing for entity resolution with MapReduce
复制标题
DOI：
10.1145/2063576.2063976
复制
发表时间：
2011-10
期刊：
影响因子：
--
通讯作者：
Lars Kolb;Andreas Thor;E. Rahm
中科院分区：
文献类型：
--
作者：
Lars Kolb;Andreas Thor;E. Rahm
基于 MapReduce 的复杂数据密集型任务实现的有效性和可扩展性取决于映射和化简任务之间数据的均匀重新分配。在存在倾斜数据的情况下，需要复杂的重新分配方法来实现所有并行执行的reduce 任务之间的负载平衡。针对实体解析阻塞的复杂问题，我们提出了BlockSplit，一种支持阻塞技术的负载均衡方法，以减少实体解析的搜索空间。对真实云基础设施的评估显示了所提出方法的价值和有效性。
The effectiveness and scalability of MapReduce-based implementations of complex data-intensive tasks depend on an even redistribution of data between map and reduce tasks. In the presence of skewed data, sophisticated redistribution approaches thus become necessary to achieve load balancing among all reduce tasks to be executed in parallel. For the complex problem of entity resolution with blocking, we propose BlockSplit, a load balancing approach that supports blocking techniques to reduce the search space of entity resolution. The evaluation on a real cloud infrastructure shows the value and effectiveness of the proposed approach.