Elements: Kingfisher: Storage Management for Data Federations

点击翻译按钮获取中文摘要

A cornerstone of translating the raw capacity of a distributed system into an effective source of shared computing power is the methodical management of all the resources. Historically, previous work to manage computations on shared systems across multiple users and workloads has focused on processing resources: CPUs, GPUs, and RAM (main memory). As the amount of research data that needs to be produced and processed is ever increasing, the Kingfisher framework addresses an important missing link in the chain of distributed capacity management: data storage space. Kingfisher provides abstractions that offer researchers and system administrators semantics for managing the sharing of storage spaces across competing workloads. Once storage can be effectively shared, providers across the nation can allocate storage to the commons through entities such as the OSG Consortium’s Open Science Data Federation, enabling researchers to harness the power of distributed computing in support of data intensive applications.This project focuses on the development and integration of the first element of the Kingfisher framework - the LotManager library. To advance the national CI ecosystem, the storage space management capabilities of the LotManager are integrated with an array of distributed software tools. These tools include the widely adopted HTCondor Software Suite and the XRootD software framework which are used by the OSG Consortium to provide data delivery to the nation’s Science and Engineering (S&E) community. The LotManager is designed to facilitate the sharing of a finite amount of storage capacity across users and projects. Sharing of computing capacity, across administrative and physical boundaries, has been proven to advance scientific discovery via Distributed High Throughput Computing (dHTC) at the national and international scales. Such work indicates the value that effective federation of storage capacity can bring to the NSF S&E community. The storage management abstractions and capabilities embodied by LotManager are developed in the context of, and with the feedback from, the OSG's production compute and storage services with real-life experimental data. Consequently, these abstractions and principles result in translational computer science research generally applicable to multi-tenant storage systems beyond the specific system developed in this project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.