Enabling Efficient and Reliable Transition from Replication to Erasure Coding for Clustered File Systems
Enabling Efficient and Reliable Transition from Replication to Erasure Coding for Clustered File Systems
复制标题
DOI：
10.1109/tpds.2017.2678505
复制
发表时间：
2017-09
期刊：
影响因子：
--
通讯作者：
Runhui Li;Yuchong Hu;P. Lee
中科院分区：
文献类型：
--
作者：
Runhui Li;Yuchong Hu;P. Lee
为了平衡性能和存储效率，现代集群文件系统（CFS）通常首先使用随机复制（即，在随机选择的节点上分布副本），然后用纠删编码对复制的数据进行编码。我们认为，随机复制，而被普遍使用，不考虑纠删编码，因此将提高性能和可用性的问题，随后的编码操作。我们提出编码感知复制，仔细放置副本，以便（i）避免在编码过程中跨机架下载数据块，（ii）保持可用性，编码后没有数据重新定位，以及（iii）保持负载平衡，在随机复制。我们在HDFS上实现编码感知复制，并通过测试实验表明，它实现了显着的编码吞吐量增益随机复制。我们还表明，通过离散事件模拟编码感知复制仍然有效的各种参数选择下，在大规模的设置。我们进一步表明，编码感知复制均匀地分布在随机复制副本。
To balance performance and storage efficiency, modern clustered file systems (CFSes) often first store data with random replication (i.e., distributing replicas across randomly selected nodes), followed by encoding the replicated data with erasure coding. We argue that random replication, while being commonly used, does not take into account erasure coding and hence will raise both performance and availability issues to the subsequent encoding operation. We propose encoding-aware replication, which carefully places the replicas so as to (i) avoid cross-rack downloads of data blocks during encoding, (ii) preserve availability without data relocation after encoding, and (iii) maintain load balancing as in random replication. We implement encoding-aware replication on HDFS, and show via tested experiments that it achieves significant encoding throughput gains over random replication. We also show via discrete-event simulations that encoding-aware replication remains effective under various parameter choices in a large-scale setting. We further show that encoding-aware replication evenly distributes replicas as in random replication.