A Scalable On-Line Associative Deep Store

可扩展的在线关联深度存储深度存储架构通过基于内容存储类似数据和使用差分压缩来压缩文件，解决了能够以高压缩率在廉价的磁性硬盘上存储数据的问题。深度存储文件检索性能消除了对磁带等传统归档存储介质的需求。深度存储从文件中计算数据指纹，然后进行摘要。通过使用基于内容的文件相似性度量将文件组织成数据簇;文件内容是不可变的，通过内容寻址并使用散列存储。研究包括开发可扩展的系统架构，开发新算法，整合现有技术，并识别和解决新问题：在非常大的语料库中搜索相似的文件以改进压缩，最大化存储吞吐量，为了吞吐量和可靠性而分布大系统，并管理数十亿个文件的文件相似性数据。提高了归档数据存储密度和检索性能（延迟和数据吞吐量）将影响所有计算环境，包括科学实验和模拟，商业和政府文档管理，扩展了按内容检索的知识。

A Scalable On-Line Associative Deep StoreThe deep store architecture solves the problems of being able to store data on inexpensive magnetic hard disk at high rates of compression by storing similar data based on content and compressing files using differential compression. Deep store file retrieval performance obviates the need for traditional archival storage media such as magnetic tape. The deep store computes data fingerprints, and then summaries, from files. Files are organized into data clusters by using a content-based file similarity metric; file content, which is immutable, is addressed by content and stored using hashing.The research comprises development of a scalable system architecture, development of new algorithms, integrating existing technologies, and identifying and addressing new problems: searching for similar files in a very large corpus to improve compression, maximizing storage throughput, distributing a large system for throughput and reliability, and managing file similarity data for billions of files.Improved archival data storage density and retrieval performance (latency and data throughput) will impact all computing environments including scientific experimentation and simulation, commercial and government document management, and expand knowledge of information retrieval by content.