EAGER: Data Deduplication with Consideration of Data Chunk Frequency

随着经济的全球化，业务数据需要每周7天、每天24小时可用。此外，在发生灾难时，必须尽快恢复数据，以最大限度地减少业务？财务损失。由于我们目前的互联网环境是真正分布式的，数据被多次复制和修改。因此，数据或数据的部分是高度冗余的。如何以合理的成本存储、保存和管理海量的数字数据成为一个非常具有挑战性的问题。 重复数据删除技术被广泛应用于日常生活中的数据驱动应用程序，并被广泛部署用于消除数据冗余，以便可以轻松管理和更好地保存大量数据。然而，对这一问题的理论认识在很大程度上仍然缺失。在这个项目中，PI计划首先调查重复数据消除的几个基本问题，然后使用这些新的见解来设计更高效的数据新算法，以实现高效的数据归档和备份。预期的原型系统将是开放源码的，并提供给其他人。拟议的项目将通过吸收工业界的投入、在本科生和研究生两级开发新课程以及强调学生群体的多样性来加强教育进程。重复数据消除的效率对长期数据保存和管理现有大量数字数据的便利性都有很大影响。从大规模模拟和建模到电子病历，再到保存和管理我们的个人数据，许多关键应用都依赖于保存和管理，从而增强了这项工作的影响力

With the globalization of the economy, business data needs to be available twenty-four hours a day, seven days a week. Furthermore, in the event of a disaster, the data must be restored as quickly as possible to minimize the business? financial loss. Since our current Internet environment is truly distributed, data are copied and revised many times. Therefore, the data or portions of the data are highly redundant. How to store, preserve and manage the enormous amount of digital data with a reasonable cost become very challenging. Data de-duplication is used to support many data driven applications in our daily life, and is widely deployed for data redundancy elimination so that the huge volumes of data can be easily managed and better preserved. However, the theoretical understanding of the problem is still largely missing. In this project, the PI plans first to investigate several fundamental issues of data de-duplication and then use these new insights to design more efficient data new algorithms for efficient data archiving and backup. The anticipated prototype system will be open source and made available to others. The proposed project will enhance the education process by bringing input from industry, developing new courses at both undergraduate and graduate levels and emphasizing the diversity of the student population. The efficiency of data de-duplication has a great impact on both long-term data preservation and ease of managing the existing huge volume of digital data. Many crucial applications from large scale simulation and modeling to electronic patient records to preserving and managing our personal data depend on both preservation and management, enhancing the impact of this work