Collaborative Research: Scalable Data Management Using Metadata and Provenance

这个项目正在开发识别和管理文件的新技术，用基于内容和元数据的搜索访问取代树状结构的文件名。通过利用现有的搜索工作和识别存储数据量的爆炸式增长，该项目使用户能够基于文件的内容、用户分配的标签、系统元数据和来源（关于文件起源的信息），以自然和直观的方式查找和访问他们的数据。这项研究的目标是高端计算（HEC）用户，他们管理着由测量设备、实验或科学工作流程生成的数十亿个文件。所开发的技术和系统也适用于通用计算。实现这一目标需要在几个领域取得进展。首先，该项目正在设计和开发快速、可扩展的机制，以收集、维护和索引HEC应用程序和用户生成的大量元数据和来源。这个项目也在探索对图结构进行操作的搜索算法，使用户能够在他们当前工作空间“附近”找到文件。为了使用户能够访问此功能，该项目正在开发一种新的“语言”，以促进用户所需的搜索类型。

Abstract This project is developing new techniques for identifying and managing files, replacing tree-structured file names with content- and metadata- based search access. By leveraging existing work in search and recognizing the explosion in the volume of data stored, this project enables users to find and access their data in natural and intuitive ways, based on the files' contents, tags the user has assigned, system metadata, and provenance (information about the file's origins). This research targets high-end computing (HEC) users, who manage billions of files generated by measurement devices, experimentation, or scientific workflows. The techniques and system developed are also applicable to general-purpose computing. Realizing this goal requires advances in several areas. First, the project is designing and developing fast, scalable mechanisms to gather, maintain and index the large volume of metadata and provenance that HEC applications and users generate. This project is also exploring search algorithms that operate on graph structures, enabling users to find files "near" their current workspace. To enable users to access this functionality, the project is developing a new "language" that facilitates the kind of searches that users need.