Boafft: Distributed Deduplication for Big Data Storage in the Cloud
Boafft: Distributed Deduplication for Big Data Storage in the Cloud
复制标题
DOI：
10.1109/tcc.2015.2511752
复制
发表时间：
2020-10
影响因子：
6.5
通讯作者：
Shengmei Luo;Guangyan Zhang;Chengwen Wu;S. Khan;Keqin Li
中科院分区：
文献类型：
--
作者：
Shengmei Luo;Guangyan Zhang;Chengwen Wu;S. Khan;Keqin Li
随着数据在数据中心内逐渐增长，云存储系统在节省存储容量和提供在可接受的时间范围内移动大数据所需的能力方面不断面临挑战。在本文中，我们提出了Boafft，一个云存储系统与分布式重复数据删除。Boafft使用多个数据服务器并行消除重复数据，以最小的重复数据消除率损失实现可扩展的吞吐量和容量。首先，Boafft使用基于数据相似性的高效数据路由算法，通过快速识别存储位置来减少网络开销。其次，Boafft在每个数据服务器中维护内存中的相似性索引，这有助于避免大量的随机磁盘读取和写入，这反过来又加速了本地重复数据删除。第三，Boafft根据访问频率在每个数据服务器上构造热指纹缓存，从而提高数据去重率。我们与EMC的状态路由算法的比较分析表明，Boafft可以提供一个相对较高的重复数据消除率与较低的网络带宽开销。此外，Boafft更好地利用了存储空间，具有更高的读/写带宽和良好的负载平衡。
As data progressively grows within data centers, the cloud storage systems continuously facechallenges in saving storage capacity and providing capabilities necessary to move big data within an acceptable time frame. In this paper, we present the Boafft, a cloud storage system with distributed deduplication. The Boafft achieves scalable throughput and capacity usingmultiple data servers to deduplicate data in parallel, with a minimal loss of deduplication ratio. Firstly, the Boafft uses an efficient data routing algorithm based on data similarity that reduces the network overhead by quickly identifying the storage location. Secondly, the Boafft maintains an in-memory similarity indexing in each data server that helps avoid a large number of random disk reads and writes, which in turn accelerates local data deduplication. Thirdly, the Boafft constructs hot fingerprint cache in each data server based on access frequency, so as to improve the data deduplication ratio. Our comparative analysis with EMC's stateful routing algorithm reveals that the Boafft can provide a comparatively high deduplication ratio with a low network bandwidth overhead. Moreover, the Boafft makes better usage of the storage space, with higher read/write bandwidth and good load balance.