CC* Data Storage: Software Defined Storage for Composable and HPC Workflows

大学校园中的高性能计算(HPC)资源传统上提供多层存储(临时、项目和归档)，以适应科学计算工作流的各种I/O模式、数据大小和保留要求。传统上，这些都是由基于POSIX的存储系统和自动磁带库提供的。最近园区级采用了像Kubernetes这样的原生云技术，产生了科学计算工作流，需要额外的数据访问方法，而传统方法是不可能的。园区存储产品必须不断发展，以支持非POSIX数据访问方法和专门构建的存储设备的配置，在提供传统的基于POSIX的存储类型的同时提供额外的存储类型，以实现整体产品。该项目通过部署中央、共享、数PB的Cave分布式存储系统来缩小园区存储和云存储之间的差距，该系统可：1)通过在园区级别建立软件定义存储即服务(STaaS)模型来增强园区共享存储能力，跨可组合资源和HPC资源提供对数据块、对象和文件系统存储的统一访问和按需调配。2)通过创新的存储基础设施支持Science领域，在仪器设备、园区可组合和HPC资源与公共云之间提供新的数据访问方法。3)通过让本科生参与存储基础设施的部署、运营和集成，促进教育和劳动力发展，并支持普渡大学校园内研究人员使用的云原生应用程序以及体验式和住宅学习计划。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

High Performance Computing (HPC) resources on university campuses have traditionally provided multiple tiers of storage (scratch, project and archive) to accommodate the various I/O patterns, data sizes and retention requirements of scientific computing workflows. Traditionally, these have been provided by POSIX based storage systems and robotic tape libraries. Recent campus level adoption of cloud native technologies like Kubernetes have produced scientific computing workflows that require additional data access methods not possible via traditional methods. Campus storage offerings must evolve to support non-POSIX data access methods and provisioning of purpose-built storage devices, providing additional storage types alongside traditional POSIX based ones, for a holistic offering. This project closes the gap between campus and cloud storage through the deployment of a central, shared, multi-petabyte Ceph distributed storage system that: 1) Enhances campus shared storage capabilities by establishing a model for Software Defined Storage as a Service (STaaS) at the campus level, offering unified access to and on-demand provisioning of block, object and filesystem storage across composable and HPC resources. 2) Supports Science Domains via innovative storage infrastructure, providing new data access methods between instrumentation, campus composable and HPC resources and the public cloud. 3) Enables education and workforce development by engaging with undergraduate students in deployment, operation and integration of storage infrastructure and supports cloud native applications used by researchers and experiential and residential learning programs on the Purdue University campus.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.