CSR-PDOS-Content-Searchable Storage for Feature-Rich Data

在过去的20年里，存储容量和数据量每18个月翻一番。构建下一代存储系统的一个关键挑战是管理大量特征丰富的（非文本）数据，这些数据已经主导了越来越多的数字信息。比较有噪声的、特征丰富的数据需要快速的相似度匹配而不是精确匹配，因此探索这样的数据需要相似度搜索而不是精确搜索。当前的文件系统是为命名文本文件设计的；它们没有管理功能丰富的数据的机制。到目前为止，还没有实用的存储系统能够对噪声、高维数据进行相似度搜索，也没有有效的相似度搜索索引引擎设计。本研究通过研究如何设计和实现一个内容可寻址和可搜索存储（CASS）系统来管理和探索各种特征丰富的数据，从而解决了这一问题。该系统包括一个内置的相似度搜索引擎，用于通用、噪声、高维元数据，使用紧凑的数据结构和新颖的索引方法。研究还将开发音频、图像和基因组数据的分割方法和特征提取方法，并开发相似性搜索基准和评估CASS系统。本研究将促进存储系统设计领域的知识和理解，如数据结构、机制和api，用于管理、搜索和探索嘈杂、高维特征丰富的数据。这项研究将加速下一代存储系统的发展，这将彻底改变如何访问、搜索、探索和管理许多学科中大量特征丰富的数据。

Storage capacity and data volume have been doubling every 18 months during the pasttwo decades. A key challenging issue in building next-generation storage systems is tomanage massive amounts of feature-rich (non-text) data, which has dominated theincreasing volume of digital information. Comparing noisy, feature-rich data requires fast similarity match instead of exact match, and thus exploring such data requires similaritysearch instead of exact search. Current file systems are designed for named text files;they do not have mechanisms to manage feature-rich data. To date, there is no practicalstorage system with the ability to do similarity search for noisy, high-dimensional dataand there is no index engine design for efficient similarity search. This researchaddresses this problem by studying how to design and implement a content-addressableand -searchable storage (CASS) system to manage and explore diverse feature-rich data.The system includes a built-in similarity search engine for general-purpose, noisy, highdimensionalmetadata using compact data structures and novel indexing methods. Theresearch will also develop segmentation methods and feature extraction methods foraudio, image and genomic data, and develop similarity search benchmarks and toevaluate the CASS system.This research will advance knowledge and understanding in the area of storage systemdesigns such as data structures, mechanisms, and APIs for managing, searching andexploring noisy, high-dimensional feature-rich data. The research will accelerate thedevelopment of next-generation storage systems which will revolutionize how to access,search, explore and manage massive amounts of feature-rich data in many disciplines.