CC* Data Storage: Institutional Storage for the University of Notre Dame (NDStore)

该项目装备了圣母大学研究计算中心（ND CRC）及其所有圣母大学学院和部门的科学用户，通过收购称为NDStore的机构存储来实现社会、物理科学和工程领域的变革性研究。NDStore的主要受益者是利用圣母大学各种研究核心的研究人员，如基因组学和生物信息学核心和圣母大学综合成像设施，以及来自大学中心和研究所的其他研究人员，如数据与社会研究所，以及通过开放科学网格的更广泛的国家社区。这些不同的设施和研究人员每年一起产生数百tb的数据，它们使专家用户能够解决当今世界上最复杂的研究问题。NDStore提供的主要功能加速了现有的研究，否则将受到存储能力不足的限制。它们还可以实现以前无法实现的规模的完整数据生命周期，实现新的国家数据密集型合作，并孵化新的研究项目。NDStore为圣母大学带来了额外的2pb存储容量，用于数据操作、管理和长期保存，以及250tb的快速刮刮存储，用于机器学习相关的工作负载。NDStore是基于开源ceph存储集群标准的高可用性解决方案。它的设计具有灵活性，以满足研究人员在其研究的不同阶段的需要。NDStore为研究人员提供了一个明显的好处，这些研究人员使用核心设施中的各种仪器生成数据，需要在数据与社区共享之前将数据传输到他们的主目录进行分析和管理。在该项目获得资助之前，ND CRC为每个学院实验室提供的存储容量不能满足大多数用户处理来自核心设施（如显微镜、测序机或其他台式设备）仪器的大数据。此外，ND CRC的高性能刮刮存储系统在高性能计算和机器学习工作负载之间共享；通常，混合这些工作负载会导致性能瓶颈，对Notre Dame的所有存储系统用户产生负面影响。NDStore帮助圣母大学为机器学习工作负载创建一个独立的刮刮系统。该项目智力价值的另一个重要方面是，CRC有机会部署NDStore，从而支持圣母大学研究企业的整个数据生命周期。ND的研究数据用例是高度多样化、复杂和异构的。它们在捕获的数据类型、使用的科学仪器、进行的数据处理和分析、数据共享和使用的政策和方法以及实验室内部的网络基础设施相关知识方面存在差异。数据生命周期阶段包括：1)数据捕获；2)仪器附近的初始加工；3)数据中心或云的中央处理；4)数据存储、管理和归档；5)数据访问、传播和可视化。在部署NDStore之前，Notre Dame的基础设施只能充分支持第1-3阶段和第5阶段，对第4阶段的支持非常少。NDStore填补了这一空白。NDStore还将整合到由大学和ND CRC众多教师主持的课堂和本科生实习项目中。通过用户培训、面向本科生的研究体验、面向高中生的大学预科课程、研讨会、实习和体验式培训项目，ND CRC将确保NDStore对当地和全国学术界产生最广泛的影响。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project equips the Center for Research Computing at the University of Notre Dame (ND CRC) and its scientific users across all Notre Dame colleges and departments to enable transformative research in social and physical sciences and engineering domains through the acquisition of institutional storage called NDStore. Major beneficiaries of NDStore are researchers utilizing various research cores at Notre Dame, such as the Genomics and Bioinformatics Core and the Notre Dame Integrated Imaging Facility, as well as other researchers from University Centers and Institutes, such as the Institute for Data and Society, in addition to the broader national community via the Open Science Grid. Together, these different facilities and researchers generate hundreds of terabytes of data per year, and they enable expert users to address the most complex research problems of today’s world. The major capabilities provided by NDStore accelerate existing research otherwise throttled by insufficient storage capability. They also enable full data lifecycle at previously inaccessible scales, enable new national data-intensive collaborations, and incubate new research projects. NDStore brings to Notre Dame an additional 2 petabytes of storage capacity for data manipulation, curation, and long-term preservation, as well as 250 terabytes of fast scratch storage for machine learning-related workloads. NDStore is a highly available solution based on an open-source Ceph-based storage clustering standard. It was designed with the flexibility to meet the needs of researchers at various stages of their research. NDStore provides a clear benefit to researchers who generate data with various instruments in core facilities and need to transfer the data to their home directories for analysis and curation before the data is shared with their communities. Before this project was funded, the amount of storage provided by ND CRC to each faculty lab was not satisfactory for most of the users dealing with large data coming from instruments at core facilities, such as microscopes, sequencing machines, or other benchtop devices. In addition, ND CRC’s high-performance scratch storage system has been shared between high-performance computing and machine learning workloads; very often, mixing these workloads led to performance bottlenecks, negatively impacting all of the storage system users at Notre Dame. NDStore helps Notre Dame create an independent scratch system for machine learning workloads.Another important aspect of the intellectual merit of this project is the opportunity for the CRC to deploy NDStore in such a way that the entire data lifecycle at Notre Dame’s research enterprise is supported. Research data use cases at ND are highly diverse, complex, and heterogeneous. They differ in types of data captured, scientific instruments used, data processing and analyses conducted, policies and methods for data sharing and use, and, internal to the lab, cyberinfrastructure-related knowledge. Data life cycle stages include: 1) data capture; 2) initial processing near the instrument(s); 3) central processing at data centers or clouds; 4) data storage, curation, and archiving; and 5) data access, dissemination, and visualization. Until NDStore was deployed, Notre Dame infrastructure could adequately support only stages 1-3 and 5, with very minimal support for stage 4. NDStore fills this gap. NDStore will also be integrated into classroom and undergraduate internship programs hosted by numerous faculty in colleges and ND CRC. Through user training, research experience for undergraduates, pre-college programs for high school students, workshops, internships, and experiential training programs, ND CRC will ensure that NDStore has the broadest possible impact on the local and national academic community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.