III: Small: Low-Cost Deduplication and Search for Versioned Datasets

组织和公司经常存档大量版本化的数字数据集。在开发数据保存、电子发现和法规遵从性所需的集成档案和搜索支持方面，存在着研究挑战和机遇。由于版本化的数据集包含高度重复的内容，重复数据删除可以将存储需求减少一个数量级或更多；然而，这样的优化是资源密集型的。重复数据删除后，版本化数据的倒排索引结构变得复杂，查找相关结果的成本较高。该项目将研究紧凑归档和索引的低成本解决方案，并开发用于搜索版本化数据集的高效算法和系统技术。它还将考虑归档数据可以存储在不受信任的服务器环境中，并研究搜索效率和隐私保护方面的权衡。开发的解决方案将为涉及大规模版本数据管理和搜索的应用程序用户带来显著的计算和存储成本优势。开发的软件将向研究团体公开。这项研究工作将与一项教育计划相结合，其中包括研究指导、教学改进和外展活动。该项目将重点研究管理大型版本数据集的集成归档和搜索支持的关键挑战和成本敏感技术方面。主要任务包括高效的软件架构和优化，用于检测云集群架构上的重复内容，使用混合索引结构进行快速多阶段搜索，以利用内容相似度和查询特征，以及具有顶级结果排序的高效隐私保护框架。

Organizations and companies often archive high volumes of versioned digital datasets. There are research challenges and opportunities for developing integrated archival and search support needed for data preservation, electronic discovery, and regulatory compliance. Since versioned datasets contain highly repetitive content, deduplication can reduce the storage demand by an order of magnitude or more; however such an optimization is resource-intensive. After deduplication, the structure of an inverted index for versioned data becomes complex and it is expensive to search relevant results. This project will study low-cost solutions for compact archiving and indexing and develop efficient algorithms and systems techniques for searching versioned datasets. It will also consider that the archived data can be stored in an untrusted server environment and investigate tradeoffs in efficiency and privacy-preservation for search. The developed solutions will bring significant computing and storage cost advantages for application users involving large-scale versioned data management and search. The developed software will be made public for research communities. The research effort will be integrated with an educational plan containing research mentoring, instruction improvement, and outreach activities.This project will be focused on studying key challenges and cost-sensitive technical aspects in integrated archival and search support for managing large versioned datasets. The main tasks include efficient software architecture and optimization for detecting duplicated content on a cloud cluster architecture, fast multi-phase search with a hybrid index structure to exploit content similarity and query characteristics, and an efficient privacy-preserving framework with top result ranking.