A Scalable On-Line Associative Deep Store

可扩展的在线关联深度存储

• 批准号: 0310888
• 负责人: Darrell Long
• 金额: $ 20.5万
• 依托单位: University of California-Santa Cruz
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2003
• 资助国家: 美国
• 起止时间: 2003-07-15 至 2007-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0310888&HistoricalAwards=false
• 关键词: Scalable; Line; Associative; Deep; Store

A Scalable On-Line Associative Deep StoreThe deep store architecture solves the problems of being able to store data on inexpensive magnetic hard disk at high rates of compression by storing similar data based on content and compressing files using differential compression. Deep store file retrieval performance obviates the need for traditional archival storage media such as magnetic tape. The deep store computes data fingerprints, and then summaries, from files. Files are organized into data clusters by using a content-based file similarity metric; file content, which is immutable, is addressed by content and stored using hashing.The research comprises development of a scalable system architecture, development of new algorithms, integrating existing technologies, and identifying and addressing new problems: searching for similar files in a very large corpus to improve compression, maximizing storage throughput, distributing a large system for throughput and reliability, and managing file similarity data for billions of files.Improved archival data storage density and retrieval performance (latency and data throughput) will impact all computing environments including scientific experimentation and simulation, commercial and government document management, and expand knowledge of information retrieval by content.

可扩展的在线关联深度存储深度存储架构通过基于内容存储相似的数据和使用差分压缩压缩文件，解决了能够以高压缩率在廉价的磁性硬盘上存储数据的问题。深度存储的文件检索性能消除了传统的档案存储介质，如磁带的需要。深度存储计算数据指纹，然后从文件中总结。使用基于内容的文件相似度度量将文件组织成数据簇；文件内容是不可变的，由内容寻址并使用散列存储。该研究包括可扩展系统架构的开发、新算法的开发、集成现有技术以及识别和解决新问题：在非常大的语料库中搜索相似文件以改进压缩、最大化存储吞吐量、分布大型系统以提高吞吐量和可靠性，以及管理数十亿文件的文件相似性数据。改进的档案数据存储密度和检索性能（延迟和数据吞吐量）将影响所有计算环境，包括科学实验和模拟、商业和政府文档管理，并扩展按内容检索信息的知识。