CSR-PDOS-Content-Searchable Storage for Feature-Rich Data

在过去的二十年里，存储容量和数据量每18个月翻一番。构建下一代存储系统的一个关键挑战性问题是管理大量功能丰富的（非文本）数据，这些数据主导了不断增长的数字信息量。比较嘈杂的、特征丰富的数据需要快速相似性匹配而不是精确匹配，因此探索这些数据需要相似性搜索而不是精确搜索。当前的文件系统是为命名文本文件设计的;它们没有管理功能丰富的数据的机制。到目前为止，还没有一个实用的存储系统能够对有噪声的高维数据进行相似性搜索，也没有索引引擎设计用于有效的相似性搜索。本研究通过研究如何设计和实现一个内容可寻址和可搜索的存储系统（卡斯）来管理和探索各种特征丰富的数据，该系统包括一个内置的相似性搜索引擎，用于使用紧凑的数据结构和新颖的索引方法来处理通用的、有噪声的、高维的元数据。本研究还将开发音频、图像和基因组数据的分割方法和特征提取方法，开发相似性搜索基准并评估卡斯系统。本研究将推进存储系统设计领域的知识和理解，如数据结构、机制和API，用于管理、搜索和探索有噪声的高维特征丰富数据。该研究将加速下一代存储系统的开发，这将彻底改变如何访问，搜索，探索和管理许多学科中大量功能丰富的数据。

Storage capacity and data volume have been doubling every 18 months during the pasttwo decades. A key challenging issue in building next-generation storage systems is tomanage massive amounts of feature-rich (non-text) data, which has dominated theincreasing volume of digital information. Comparing noisy, feature-rich data requires fast similarity match instead of exact match, and thus exploring such data requires similaritysearch instead of exact search. Current file systems are designed for named text files;they do not have mechanisms to manage feature-rich data. To date, there is no practicalstorage system with the ability to do similarity search for noisy, high-dimensional dataand there is no index engine design for efficient similarity search. This researchaddresses this problem by studying how to design and implement a content-addressableand -searchable storage (CASS) system to manage and explore diverse feature-rich data.The system includes a built-in similarity search engine for general-purpose, noisy, highdimensionalmetadata using compact data structures and novel indexing methods. Theresearch will also develop segmentation methods and feature extraction methods foraudio, image and genomic data, and develop similarity search benchmarks and toevaluate the CASS system.This research will advance knowledge and understanding in the area of storage systemdesigns such as data structures, mechanisms, and APIs for managing, searching andexploring noisy, high-dimensional feature-rich data. The research will accelerate thedevelopment of next-generation storage systems which will revolutionize how to access,search, explore and manage massive amounts of feature-rich data in many disciplines.