Deduplication-aware Systems for Cost-efficient Cloud Storage

云存储系统是包括大数据分析和物联网（IoT）在内的新兴应用的重要基础设施。一个关键的挑战是如何以具有成本效益的方式实时处理大量数据。数据量和复杂性的爆炸性增长加剧了这一挑战。此外，许多云计算系统是联网的和分布式的，因此由于有限的带宽而使得存储系统管理更加复杂和昂贵。 重复数据删除是一种有效的数据减少方法，不仅通过消除重复数据来减少存储空间，而且即使在低带宽环境中也能最大限度地减少冗余数据的传输。然而，传统的去重方案在分块中具有高计算复杂度，并且用于存储块索引的存储开销大，因此无法提供实时和成本有效的存储服务。 该研究计划旨在解决针对大数据应用的云存储系统性能优化所面临的最重要挑战。我们将开展创新研究，以克服目前基于重复数据删除的方法的局限性。为此，我们将研究自适应多粒度重复数据删除方案，以显着减少要处理的数据量，提高整体系统性能。我们计划提出一种新的方法，重复数据删除粒度，以满足处理不同的数据规模的需求。实现技术，包括本地感知哈希，重复数据删除，压缩协同和管道调度，将在真实的云存储平台进行调查，评估和验证。

Cloud storage systems serve as an important infrastructure for emerging applications including Big Data Analytics and Internet of Things (IoT). A key challenge is how to handle a massive amount of data in real-time in a cost-efficient way. Explosive growth in the volume and complexity of data exacerbates this challenge. Further, many cloud computing systems are networked and distributed, thus making storage system management more complex and costly due to the limited bandwidth. Data deduplication is an efficient data reduction approach that not only reduces storage space by eliminating duplicate data but also minimizes the transmission of redundant data even in low-bandwidth environments. However, conventional deduplication schemes suffer from high computation complexity in chunking and large storage overhead for storing block indices, thus failing to offer real-time and cost-efficient storage services. This research program aims to address the most important challenges facing the performance optimization of cloud storage systems targeting big data applications. We will conduct innovative research to overcome the current limitations of deduplication-based methodologies. To this end, we will investigate adaptive multi-granularity deduplication schemes to significantly reduce the amounts of data to be processed and improve the overall system performance. We plan to propose a new methodology in deduplication granularities to meet the needs of handling different data scales. Implementation techniques, including locality-aware hashing, deduplication, and compression synergization and pipeline scheduling, will be investigated, evaluated, and validated in real cloud storage platform.