III: Small: Low-Cost Deduplication and Search for Versioned Datasets

组织和公司经常存档大量版本化数字数据集。有研究的挑战和机遇，开发集成的存档和搜索支持所需的数据保存，电子发现，和法规遵从性。由于版本化数据集包含高度重复的内容，因此重复数据删除可以将存储需求降低一个数量级或更多;然而，这种优化是资源密集型的。在重复数据删除之后，版本化数据的倒排索引的结构变得复杂，并且搜索相关结果的成本很高。 该项目将研究用于紧凑存档和索引的低成本解决方案，并开发用于搜索版本化数据集的有效算法和系统技术。它还将考虑归档数据可以存储在不受信任的服务器环境中，并调查搜索效率和隐私保护的权衡。开发的解决方案将为涉及大规模版本化数据管理和搜索的应用程序用户带来显着的计算和存储成本优势。开发的软件将向研究界公开。研究工作将与一项教育计划相结合，该计划包括研究指导、教学改进和推广活动。该项目将侧重于研究管理大型版本数据集的综合存档和搜索支持方面的关键挑战和成本敏感的技术问题。主要任务包括高效的软件架构和优化，用于检测云集群架构上的重复内容，使用混合索引结构进行快速多阶段搜索，以利用内容相似性和查询特征，以及具有最高结果排名的高效隐私保护框架。

Organizations and companies often archive high volumes of versioned digital datasets. There are research challenges and opportunities for developing integrated archival and search support needed for data preservation, electronic discovery, and regulatory compliance. Since versioned datasets contain highly repetitive content, deduplication can reduce the storage demand by an order of magnitude or more; however such an optimization is resource-intensive. After deduplication, the structure of an inverted index for versioned data becomes complex and it is expensive to search relevant results. This project will study low-cost solutions for compact archiving and indexing and develop efficient algorithms and systems techniques for searching versioned datasets. It will also consider that the archived data can be stored in an untrusted server environment and investigate tradeoffs in efficiency and privacy-preservation for search. The developed solutions will bring significant computing and storage cost advantages for application users involving large-scale versioned data management and search. The developed software will be made public for research communities. The research effort will be integrated with an educational plan containing research mentoring, instruction improvement, and outreach activities.This project will be focused on studying key challenges and cost-sensitive technical aspects in integrated archival and search support for managing large versioned datasets. The main tasks include efficient software architecture and optimization for detecting duplicated content on a cloud cluster architecture, fast multi-phase search with a hybrid index structure to exploit content similarity and query characteristics, and an efficient privacy-preserving framework with top result ranking.