Collaborative Research: CIF: Small: Versatile Data Synchronization: Novel Codes and Algorithms for Practical Applications

据估计，创建或复制的全球数字数据总量大约每三年翻一番。这种快速增长导致对个人、企业和科学环境中可靠和通用的数据访问的需求日益增加。为了满足这些需求，数据同步（指存储在不同主机上的不同版本数据之间保持一致性的过程）已成为数据管理的一个重要方面。然而，最先进的同步工具有明显的缺点和低效率，导致成本增加和高延迟访问。该项目旨在开发基于纠错码的具有最佳通信带宽的数据同步算法，并将同步的适用性扩大到当前工具不足的现实环境中。除了科学和技术进步之外，该项目还有可能促进诸如农村地区等宽带互联网接入有限的用户使用分布式存储系统；帮助减少与数据传输相关的能源消耗；并提供机会参与和培养本科生研究人员。该项目的目标将通过三个研究重点来实现。第一个目标是通过设计低冗余的系统编辑校正码以及高效的编码和解码算法来提高数据同步的效率。第二个重点是同步压缩数据。由于传统压缩通常会破坏相关文件之间的相似性，因此该项目将开发相互兼容的压缩和同步方法，鉴于数据压缩的流行，这些方法有可能大大扩展同步对大型数据集的使用。最后，第三个重点将从理论和实践的角度解决经常被忽视的现实世界同步约束。特别是，在一方受到通信或计算约束的情况下，将建立信息交换的边界。此外，将开发增量和自适应同步协议，以便在控制数据更新和修改的随机过程的统计数据未知时有效地同步数据。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The total volume of global digital data created or copied is estimated to double approximately every three years. This rapid growth has led to an increasing need for reliable and universal access to data in personal, enterprise, and scientific environments. To meet these requirements, data synchronization, which refers to the process of maintaining consistency between different versions of data stored on separate hosts, has become a crucial aspect of managing data. However, state-of-the-art synchronization tools have significant shortcomings and inefficiencies, resulting in increased costs and high-latency access. This project aims to develop data synchronization algorithms with optimal communication bandwidth based on error-correcting codes and to broaden the applicability of synchronization to real-world settings where current tools are inadequate. In addition to scientific and technological advances, the project has the potential to facilitate access to distributed storage systems for users with limited access to broadband Internet, such as in rural areas; help reduce energy consumption associated with data transmission; and provide opportunities to engage and train undergraduate researchers. The goals of the project will be achieved through three research thrusts. The first thrust aims to increase the efficiency of data synchronization by designing low-redundancy systematic edit-correcting codes, along with efficient encoding and decoding algorithms. The second thrust focuses on synchronizing compressed data. As conventional compression typically destroys the similarity between related files, the project will develop mutually compatible compression and synchronization methods which, given the prevalence of data compression, have the potential to significantly expand the use of synchronization for large datasets. Finally, the third thrust will address often-overlooked real-world constraints on synchronization from theoretical and practical points of view. In particular, bounds on the information exchange will be established where one party is under communication or computational constraints. Furthermore, incremental and adaptive synchronization protocols will be developed to efficiently synchronize data when the statistics of the stochastic processes governing data update and modification are unknown.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.