Usable Disk Space Control Based on Hadoop Job Features
Usable Disk Space Control Based on Hadoop Job Features
复制标题
基于Hadoop作业特征的可用磁盘空间控制
DOI：
10.1109/candarw.2019.00093
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
Yamaguchi, Saneyasu
中科院分区：
文献类型：
--
作者：
Nakagami, Makoto;Fortes, Jose A.B.;Yamaguchi, Saneyasu
Hadoop是一个用于大数据处理的开源平台。在分析超大数据的情况下，通常使用硬盘驱动器。硬盘驱动器具有不同的性能，具体取决于数据在磁盘上的位置。在其他已发表的工作中，提出了一种通过使用Hadoop作业的特征来决定数据放置位置来提高Hadoop I/O性能的方法。这种方法将整个硬盘驱动器空间分为两个区域，即外部区域和内部区域。然而，这种方法并没有积极利用每个地区最快的可用区域。在本文中，我们提出了一种新的方法来改善这种方法，积极利用最快的可用区域在每个区域。我们提出的方法与流行的Hadoop基准测试的评估表明，新方法可以提高工作性能的9%，与现有的方法相比。
Hadoop is an open-source platform for big data processing. In the case of analyzing extremely big data, hard disk drives are usually used. Hard disk drives have different performance depending on where data is placed on the disk. In other published work, an approach for improving Hadoop I/O performance by using features of Hadoop jobs to decide on the location of data placement was proposed. This approach separates the entire hard disk drive space into two areas, which are outer and inner areas. However, this approach did not actively utilize the fastest available zones in each area. In this paper, we propose a new method for improving this method by active usage of the fastest available zones in each area. Our evaluation of the proposed method with a popular Hadoop benchmark shows that the new method can improve job performance by 9% when compared with existing approach.