CC* Data Storage: Software Defined Storage for Composable and HPC Workflows

传统上，大学校园中的高性能计算（HPC）资源提供了多层存储（临时存储、项目存储和存档存储），以适应科学计算工作流的各种I/O模式、数据大小和保留需求。传统上，这些是由基于POSIX的存储系统和机器人磁带库提供的。最近校园级采用Kubernetes等云原生技术产生了科学计算工作流程，需要通过传统方法无法实现的额外数据访问方法。校园存储产品必须不断发展，以支持非POSIX数据访问方法，并提供专用存储设备，除了传统的基于POSIX的存储设备外，还提供额外的存储类型，以实现整体产品。该项目通过部署中央、共享、多pb的Ceph分布式存储系统，缩小了校园与云存储之间的差距，该系统：1)通过在校园层面建立软件定义存储即服务（STaaS）模型，提供跨可组合和高性能计算资源的块、对象和文件系统存储的统一访问和按需供应，增强了校园共享存储能力。2)通过创新的存储基础设施支持科学领域，在仪器仪表、校园可组合和HPC资源与公共云之间提供新的数据访问方法。3)通过与本科生一起参与存储基础设施的部署、运营和集成，支持研究人员使用的云原生应用程序，以及普渡大学校园内的体验式和住宿式学习项目，促进教育和劳动力发展。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

High Performance Computing (HPC) resources on university campuses have traditionally provided multiple tiers of storage (scratch, project and archive) to accommodate the various I/O patterns, data sizes and retention requirements of scientific computing workflows. Traditionally, these have been provided by POSIX based storage systems and robotic tape libraries. Recent campus level adoption of cloud native technologies like Kubernetes have produced scientific computing workflows that require additional data access methods not possible via traditional methods. Campus storage offerings must evolve to support non-POSIX data access methods and provisioning of purpose-built storage devices, providing additional storage types alongside traditional POSIX based ones, for a holistic offering. This project closes the gap between campus and cloud storage through the deployment of a central, shared, multi-petabyte Ceph distributed storage system that: 1) Enhances campus shared storage capabilities by establishing a model for Software Defined Storage as a Service (STaaS) at the campus level, offering unified access to and on-demand provisioning of block, object and filesystem storage across composable and HPC resources. 2) Supports Science Domains via innovative storage infrastructure, providing new data access methods between instrumentation, campus composable and HPC resources and the public cloud. 3) Enables education and workforce development by engaging with undergraduate students in deployment, operation and integration of storage infrastructure and supports cloud native applications used by researchers and experiential and residential learning programs on the Purdue University campus.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.