Cluster and Single-Node Analysis of Long-Term Deduplication Patterns
Cluster and Single-Node Analysis of Long-Term Deduplication Patterns
复制标题
DOI：
10.1145/3183890
复制
发表时间：
2018-05
期刊：
影响因子：
--
通讯作者：
Zhen Sun;G. Kuenning;Sonam Mandal;Philip Shilane;Vasily Tarasov;Nong Xiao;E. Zadok
中科院分区：
文献类型：
--
作者：
Zhen Sun;G. Kuenning;Sonam Mandal;Philip Shilane;Vasily Tarasov;Nong Xiao;E. Zadok
重复数据删除在基于磁盘的备份系统中已变得至关重要，但对备份工作负载的长期研究却很少。过去的大多数研究要么是小型静态快照，要么只涵盖很短的一段时间，不能代表备份系统如何随时间演变。在本文中，我们首先从共享用户文件系统收集了 21 个月的数据；涵盖 33 个用户和 4,000 多个快照。然后，我们分析了数据集，跨两个维度检查了各种基本特征：单节点重复数据删除和集群重复数据删除。对于单节点重复数据删除分析，我们的主要关注点是个人用户数据。尽管我们所有用户的角色和行为明显相似，但我们发现他们的重复数据删除率存在显着差异。此外，一些用户与其他人共享的数据的重复数据删除率远高于平均水平。对于集群重复数据删除分析，我们实施了七种已发布的数据路由算法，并对它们在重复数据删除率、负载分布和通信开销方面的性能进行了详细比较。我们发现，按文件路由比按超级块（多个连续块）路由实现了更高的重复数据删除率，但它也导致了高数据倾斜（节点间空间使用不平衡）。我们还发现，大的分块大小更适合集群重复数据删除，因为它们可以显着减少数据路由开销，而它们对重复数据删除率的负面影响很小并且可以接受。我们从单节点和集群重复数据删除分析中得出有趣的结论，并为未来的重复数据删除系统设计提出建议。
Deduplication has become essential in disk-based backup systems, but there have been few long-term studies of backup workloads. Most past studies either were of a small static snapshot or covered only a short period that was not representative of how a backup system evolves over time. For this article, we first collected 21 months of data from a shared user file system; 33 users and over 4,000 snapshots are covered. We then analyzed the dataset, examining a variety of essential characteristics across two dimensions: single-node deduplication and cluster deduplication. For single-node deduplication analysis, our primary focus was individual-user data. Despite apparently similar roles and behavior among all of our users, we found significant differences in their deduplication ratios. Moreover, the data that some users share with others had a much higher deduplication ratio than average. For cluster deduplication analysis, we implemented seven published data-routing algorithms and created a detailed comparison of their performance with respect to deduplication ratio, load distribution, and communication overhead. We found that per-file routing achieves a higher deduplication ratio than routing by super-chunk (multiple consecutive chunks), but it also leads to high data skew (imbalance of space usage across nodes). We also found that large chunking sizes are better for cluster deduplication, as they significantly reduce data-routing overhead, while their negative impact on deduplication ratios is small and acceptable. We draw interesting conclusions from both single-node and cluster deduplication analysis and make recommendations for future deduplication systems design.