MHDFS: A Memory-Based Hadoop Framework for Large Data Storage
MHDFS: A Memory-Based Hadoop Framework for Large Data Storage
复制标题
DOI：
10.1155/2016/1808396
复制
发表时间：
2016-05
期刊：
影响因子：
--
通讯作者：
Aibo Song;Mao-Xian Zhao;Yingying Xue;Junzhou Luo
中科院分区：
文献类型：
--
作者：
Aibo Song;Mao-Xian Zhao;Yingying Xue;Junzhou Luo
Hadoop分布式文件系统（HDFS）无疑是在机器集群上存储和处理大量数据的最流行的框架。虽然已经提出了大量的实践来提高处理效率和资源利用率，但传统的HDFS仍然受到基于磁盘的低吞吐量和I/O速率的开销的困扰。在本文中，我们试图通过开发一个名为MHDFS的基于内存的Hadoop框架来解决这个问题。首先，设计了合理分配和配置MHDFS内存资源的策略，并利用RAMFS进行框架开发。然后，我们提出了一种新的方法来处理当内存资源被过度占用时数据到磁盘的替换。设计了一种基于文件热度量的替换估计和更新算法。最后，进行了大量的实验，证明了MHDFS的有效性及其相对于传统HDFS的优势。
Hadoop distributed file system (HDFS) is undoubtedly the most popular framework for storing and processing large amount of data on clusters of machines. Although a plethora of practices have been proposed for improving the processing efficiency and resource utilization, traditional HDFS still suffers from the overhead of disk-based low throughput and I/O rate. In this paper, we attempt to address this problem by developing a memory-based Hadoop framework called MHDFS. Firstly, a strategy for allocating and configuring reasonable memory resources for MHDFS is designed and RAMFS is utilized to develop the framework. Then, we propose a new method to handle the data replacement to disk when memory resource is excessively occupied. An algorithm for estimating and updating the replacement is designed based on the metrics of file heat. Finally, substantial experiments are conducted which demonstrate the effectiveness of MHDFS and its advantage against conventional HDFS.