SS-CDC: a two-stage parallel content-defined chunking for deduplicating backup storage
SS-CDC: a two-stage parallel content-defined chunking for deduplicating backup storage
复制标题
DOI：
10.1145/3319647.3325834
复制
发表时间：
2019-05
期刊：
影响因子：
--
通讯作者：
Fan Ni;Xing Lin;Song Jiang
中科院分区：
文献类型：
--
作者：
Fan Ni;Xing Lin;Song Jiang
数据重复数据删除已被广泛用于存储系统中，以提高存储效率和I/O性能。特别是，内容定义的可变大小分解（CDC）通常用于数据删除系统中，以便其在修改后的文件中检测和删除重复数据的能力。但是，CDC算法是非常密集的，固有的顺序。通过分割文件并在每个细分市场上独立运行该算法来加速它的努力，以大幅度降级的重复比例降解。在本文中，我们提出了SS-CDC，一种两阶段并行CDC，它可以在不损害重复比率的情况下（几乎）完全并行的文件。此外，SS-CDC利用了当今处理器中可用的指令级SIMD并行性。作为一个案例研究，通过使用Intel AVX-512说明，SS-CDC始终在多核服务器上获得超线性加速。我们使用现实世界数据集的实验表明，与现有的平行CDC方法相比，该方法仅在8核处理器上实现高达7.7倍的速度，而重复数据删除率最高为40％，SS-CDC最多可以实现25.6倍加速，没有重复数据删除比。
Data deduplication has been widely used in storage systems to improve storage efficiency and I/O performance. In particular, content-defined variable-size chunking (CDC) is often used in data deduplication systems for its capability to detect and remove duplicate data in modified files. However, the CDC algorithm is very compute-intensive and inherently sequential. Efforts on accelerating it by segmenting a file and running the algorithm independently on each segment in parallel come at a cost of substantial degradation of deduplication ratio. In this paper, we propose SS-CDC, a two-stage parallel CDC, that enables (almost) full parallelism on chunking of a file without compromising deduplication ratio. Further, SS-CDC exploits instruction-level SIMD parallelism available in today's processors. As a case study, by using Intel AVX-512 instructions, SS-CDC consistently obtains superlinear speedups on a multi-core server. Our experiments using real-world datasets show that, compared to existing parallel CDC methods which only achieve up to a 7.7X speedup on an 8-core processor with the deduplication ratio degraded by up to 40%, SS-CDC can achieve up to a 25.6X speedup with no loss of deduplication ratio.