LaSA: A locality-aware scheduling algorithm for Hadoop-MapReduce resource assignment
LaSA: A locality-aware scheduling algorithm for Hadoop-MapReduce resource assignment
复制标题
DOI：
10.1109/cts.2013.6567252
复制
发表时间：
2013-05
期刊：
影响因子：
--
通讯作者：
Tseng-Yi Chen;H. Wei;Ming-Feng Wei;Ying-Jie Chen;T. Hsu;W. Shih
中科院分区：
文献类型：
--
作者：
Tseng-Yi Chen;H. Wei;Ming-Feng Wei;Ying-Jie Chen;T. Hsu;W. Shih
十年来，云计算变得越来越流行；它随着架构、软件和网络的进步一直在不断发展。Hadoop - MapReduce是一种常见的软件框架，它使用处理器的分布式集群或独立计算机处理跨大数据集的可并行化问题。云Hadoop - MapReduce可以在处理节点数量上逐步扩展。因此，Hadoop - MapReduce旨在提供一个具有强大计算能力的处理平台。网络流量一直是数据密集型计算中最重要的瓶颈，网络延迟会显著降低数据并行系统的性能。网络瓶颈是由网络带宽引起的，网络速度比磁盘数据访问慢得多。因此，良好的数据局部性可以减少网络流量并提高数据密集型高性能计算系统的性能。然而，Hadoop的调度器在资源分配方面存在数据局部性的缺陷。在本文中，我们为Hadoop - MapReduce调度器提出了一种具有局部性感知的调度算法（LaSA）。首先，我们提出了Hadoop调度器中数据干扰权重的数学模型。其次，我们提出了LaSA算法，利用数据干扰权重在Hadoop调度器中提供具有数据局部性感知的资源分配。最后，我们构建了一个由3个集群和35个虚拟机组成的实验环境来验证LaSA的性能。
Cloud computing has become more popular for a decade; it has been under continuous development with advances in architecture, software, and network. Hadoop-MapReduce is a common software framework processing parallelizable problem across big datasets using a distributed cluster of processors or stand-alone computers. Cloud Hadoop-MapReduce can scale incrementally in the number of processing nodes. Hence, the Hadoop-MapReduce is designed to provide a processing platform with powerful computation. Network traffic is always a most important bottleneck in data-intensive computing and network latency decreases significant performance in data parallel systems. Network bottleneck is caused by network bandwidth and the network speed is much slower than disk data access. So that, good data locality can reduces network traffic and increases performance in data-intensive HPC systems. However, Hadoop's scheduler has a defect of data locality in resource assignment. In this paper, we present a locality-aware scheduling algorithm (LaSA) for Hadoop-MapReduce scheduler. Firstly, we propose a mathematical model of weight of data interference in Hadoop scheduler. Secondly, we present the LaSA algorithm to use weight of data interference to provide data locality-aware resource assignment in Hadoop scheduler. Finally, we build an experimental environment with 3 cluster and 35 VMs to verify the LaSA's performance.