III: Small: Low-Cost Deduplication and Search for Versioned Datasets

组织和公司通常会存档大量的版本化数字数据集。在开发数据保存、电子发现和法规遵从性所需的综合档案和搜索支持方面，存在研究挑战和机会。由于版本化数据集包含高度重复的内容，因此重复数据消除可以将存储需求降低一个数量级或更多；然而，这种优化需要大量资源。重复数据删除后，版本化数据的倒排索引结构变得复杂，搜索相关结果的成本很高。该项目将研究压缩存档和索引的低成本解决方案，并开发用于搜索版本化数据集的有效算法和系统技术。它还将考虑将存档数据存储在不受信任的服务器环境中，并调查搜索在效率和隐私保护方面的权衡。开发的解决方案将为涉及大规模版本化数据管理和搜索的应用程序用户带来显著的计算和存储成本优势。开发的软件将向研究社区公开。这项研究工作将与一项教育计划相结合，该计划包括研究指导、改进教学和推广活动。该项目将侧重于研究综合档案和搜索支持管理大型版本化数据集方面的关键挑战和对成本敏感的技术问题。主要任务包括高效的软件体系结构和云集群体系结构上检测重复内容的优化，利用内容相似度和查询特征的混合索引结构的快速多阶段搜索，以及具有顶级结果排序的高效隐私保护框架。

Organizations and companies often archive high volumes of versioned digital datasets. There are research challenges and opportunities for developing integrated archival and search support needed for data preservation, electronic discovery, and regulatory compliance. Since versioned datasets contain highly repetitive content, deduplication can reduce the storage demand by an order of magnitude or more; however such an optimization is resource-intensive. After deduplication, the structure of an inverted index for versioned data becomes complex and it is expensive to search relevant results. This project will study low-cost solutions for compact archiving and indexing and develop efficient algorithms and systems techniques for searching versioned datasets. It will also consider that the archived data can be stored in an untrusted server environment and investigate tradeoffs in efficiency and privacy-preservation for search. The developed solutions will bring significant computing and storage cost advantages for application users involving large-scale versioned data management and search. The developed software will be made public for research communities. The research effort will be integrated with an educational plan containing research mentoring, instruction improvement, and outreach activities.This project will be focused on studying key challenges and cost-sensitive technical aspects in integrated archival and search support for managing large versioned datasets. The main tasks include efficient software architecture and optimization for detecting duplicated content on a cloud cluster architecture, fast multi-phase search with a hybrid index structure to exploit content similarity and query characteristics, and an efficient privacy-preserving framework with top result ranking.