Elements: Kingfisher: Storage Management for Data Federations

将分布式系统的原始容量转化为共享计算能力的有效来源的基石是对所有资源的系统化管理。从历史上看，以前管理跨多个用户和工作负载的共享系统上的计算的工作集中在处理资源上：CPU，GPU和RAM（主内存）。随着需要生成和处理的研究数据量不断增加，Kingfisher框架解决了分布式容量管理链中缺失的重要环节：数据存储空间。Kingfisher提供抽象，为研究人员和系统管理员提供语义，用于管理竞争工作负载之间的存储空间共享。一旦存储可以有效地共享，全国各地的供应商可以通过OSG Consortium的Open Science Data Federation等实体将存储分配给公共资源，使研究人员能够利用分布式计算的力量来支持数据密集型应用程序。该项目的重点是开发和集成Kingfisher框架的第一个元素-LotManager库。为了推进国家CI生态系统，LotManager的存储空间管理功能与一系列分布式软件工具相集成。这些工具包括广泛采用的HTCondor软件套件和XRootD软件框架，OSG联盟使用这些工具向国家的科学和工程（SE）社区提供数据交付。LotManager旨在促进用户和项目之间共享有限的存储容量。事实证明，跨越行政和物理边界共享计算能力可以通过分布式高吞吐量计算（dHTC）在国家和国际范围内推进科学发现。这些工作表明了存储容量的有效联合可以为NSF SE社区带来的价值。LotManager体现的存储管理抽象和功能是在OSG的生产计算和存储服务的环境中开发的，并得到了OSG的生产计算和存储服务的反馈，这些服务具有真实的实验数据。因此，这些抽象和原则导致翻译计算机科学研究普遍适用于多租户存储系统超出了本项目中开发的特定系统。这个奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

A cornerstone of translating the raw capacity of a distributed system into an effective source of shared computing power is the methodical management of all the resources. Historically, previous work to manage computations on shared systems across multiple users and workloads has focused on processing resources: CPUs, GPUs, and RAM (main memory). As the amount of research data that needs to be produced and processed is ever increasing, the Kingfisher framework addresses an important missing link in the chain of distributed capacity management: data storage space. Kingfisher provides abstractions that offer researchers and system administrators semantics for managing the sharing of storage spaces across competing workloads. Once storage can be effectively shared, providers across the nation can allocate storage to the commons through entities such as the OSG Consortium’s Open Science Data Federation, enabling researchers to harness the power of distributed computing in support of data intensive applications.This project focuses on the development and integration of the first element of the Kingfisher framework - the LotManager library. To advance the national CI ecosystem, the storage space management capabilities of the LotManager are integrated with an array of distributed software tools. These tools include the widely adopted HTCondor Software Suite and the XRootD software framework which are used by the OSG Consortium to provide data delivery to the nation’s Science and Engineering (S&E) community. The LotManager is designed to facilitate the sharing of a finite amount of storage capacity across users and projects. Sharing of computing capacity, across administrative and physical boundaries, has been proven to advance scientific discovery via Distributed High Throughput Computing (dHTC) at the national and international scales. Such work indicates the value that effective federation of storage capacity can bring to the NSF S&E community. The storage management abstractions and capabilities embodied by LotManager are developed in the context of, and with the feedback from, the OSG's production compute and storage services with real-life experimental data. Consequently, these abstractions and principles result in translational computer science research generally applicable to multi-tenant storage systems beyond the specific system developed in this project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.