CC* Data Storage: Institutional Storage for the University of Notre Dame (NDStore)

该项目为圣母大学研究计算中心（ND CRC）及其所有圣母大学学院和部门的科学用户提供装备，通过收购名为NDStore的机构存储，实现社会和物理科学以及工程领域的变革性研究。NDStore的主要受益者是利用Notre Dame各种研究核心的研究人员，如基因组学和生物信息学核心和Notre Dame综合成像设施，以及来自大学中心和研究所的其他研究人员，如数据与社会研究所，以及通过开放科学网格更广泛的国家社区。这些不同的设施和研究人员每年产生数百TB的数据，使专家用户能够解决当今世界最复杂的研究问题。NDStore提供的主要功能加速了现有的研究，否则会因存储能力不足而受到限制。它们还可以在以前无法访问的规模上实现完整的数据生命周期，实现新的国家数据密集型合作，并孵化新的研究项目。 NDStore为Notre Dame带来了额外的2 PB存储容量，用于数据操作，管理和长期保存，以及250 TB的快速暂存存储，用于机器学习相关的工作负载。NDStore是一个高度可用的解决方案，基于开源的Ceph存储集群标准。它的设计具有灵活性，以满足研究人员在不同研究阶段的需求。NDStore为那些在核心设施中使用各种工具生成数据的研究人员提供了明显的好处，并且需要在与社区共享数据之前将数据传输到其主目录进行分析和管理。 在此项目获得资助之前，ND CRC为每个教师实验室提供的存储量对于大多数处理来自核心设施仪器（如显微镜，测序机或其他台式设备）的大数据的用户来说并不令人满意。此外，ND CRC的高性能暂存存储系统在高性能计算和机器学习工作负载之间共享;通常，混合这些工作负载会导致性能瓶颈，对Notre Dame的所有存储系统用户产生负面影响。NDStore帮助Notre Dame为机器学习工作负载创建了一个独立的暂存系统。该项目的另一个重要方面是CRC有机会以支持Notre Dame研究企业整个数据生命周期的方式部署NDStore。ND的研究数据用例是高度多样化、复杂和异构的。它们在获取的数据类型、使用的科学仪器、进行的数据处理和分析、数据共享和使用的政策和方法以及实验室内部的网络基础设施相关知识方面有所不同。数据生命周期阶段包括：1）数据捕获; 2）仪器附近的初始处理; 3）数据中心或云的中央处理; 4）数据存储、管理和存档; 5）数据访问、传播和可视化。在NDStore部署之前，Notre Dame基础设施只能充分支持阶段1-3和5，对阶段4的支持非常少。NDStore填补了这一空白。 NDStore还将被整合到由大学和ND CRC的众多教师主办的课堂和本科生实习计划中。通过用户培训、本科生研究经验、高中生预科课程、研讨会、实习和体验式培训项目，ND CRC将确保NDStore对当地和国家学术界产生尽可能广泛的影响。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响力审查标准进行评估，被认为值得支持。

This project equips the Center for Research Computing at the University of Notre Dame (ND CRC) and its scientific users across all Notre Dame colleges and departments to enable transformative research in social and physical sciences and engineering domains through the acquisition of institutional storage called NDStore. Major beneficiaries of NDStore are researchers utilizing various research cores at Notre Dame, such as the Genomics and Bioinformatics Core and the Notre Dame Integrated Imaging Facility, as well as other researchers from University Centers and Institutes, such as the Institute for Data and Society, in addition to the broader national community via the Open Science Grid. Together, these different facilities and researchers generate hundreds of terabytes of data per year, and they enable expert users to address the most complex research problems of today’s world. The major capabilities provided by NDStore accelerate existing research otherwise throttled by insufficient storage capability. They also enable full data lifecycle at previously inaccessible scales, enable new national data-intensive collaborations, and incubate new research projects. NDStore brings to Notre Dame an additional 2 petabytes of storage capacity for data manipulation, curation, and long-term preservation, as well as 250 terabytes of fast scratch storage for machine learning-related workloads. NDStore is a highly available solution based on an open-source Ceph-based storage clustering standard. It was designed with the flexibility to meet the needs of researchers at various stages of their research. NDStore provides a clear benefit to researchers who generate data with various instruments in core facilities and need to transfer the data to their home directories for analysis and curation before the data is shared with their communities. Before this project was funded, the amount of storage provided by ND CRC to each faculty lab was not satisfactory for most of the users dealing with large data coming from instruments at core facilities, such as microscopes, sequencing machines, or other benchtop devices. In addition, ND CRC’s high-performance scratch storage system has been shared between high-performance computing and machine learning workloads; very often, mixing these workloads led to performance bottlenecks, negatively impacting all of the storage system users at Notre Dame. NDStore helps Notre Dame create an independent scratch system for machine learning workloads.Another important aspect of the intellectual merit of this project is the opportunity for the CRC to deploy NDStore in such a way that the entire data lifecycle at Notre Dame’s research enterprise is supported. Research data use cases at ND are highly diverse, complex, and heterogeneous. They differ in types of data captured, scientific instruments used, data processing and analyses conducted, policies and methods for data sharing and use, and, internal to the lab, cyberinfrastructure-related knowledge. Data life cycle stages include: 1) data capture; 2) initial processing near the instrument(s); 3) central processing at data centers or clouds; 4) data storage, curation, and archiving; and 5) data access, dissemination, and visualization. Until NDStore was deployed, Notre Dame infrastructure could adequately support only stages 1-3 and 5, with very minimal support for stage 4. NDStore fills this gap. NDStore will also be integrated into classroom and undergraduate internship programs hosted by numerous faculty in colleges and ND CRC. Through user training, research experience for undergraduates, pre-college programs for high school students, workshops, internships, and experiential training programs, ND CRC will ensure that NDStore has the broadest possible impact on the local and national academic community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.