CIF21 DIBBs: PD: Building High-Availability Data Capabilities in Data-Centric Cyberinfrastructure

该项目支持广泛的科学和工程应用中的数据相关分析。它将有助于开发可扩展的以数据为中心的网络基础设施能力，以加速跨学科和合作研究。 Exascale文件系统将成为数据存储架构研究的催化剂，并将实现新的以数据为中心的服务和功能，推动科学发现，合作和创新。 该项目解决了社会科学，经济学和生物工程等一系列社区共同面临的重大数据挑战，并将为项目参与者提供全面的培训和合作研究机会。高性能计算（HPC）集群和Hadoop集群都使用文件系统。Hadoop集群使用驻留在计算节点上的Hadoop分布式文件系统（HDFS），而HPC集群通常使用远程存储系统。尽管多年来一直致力于HPC和Hadoop集群的研究和应用开发，但这两种类型集群中的文件系统仍然面临着巨大的挑战，即实现百亿亿次计算能力。HDFS和HPC存储架构中的集中式数据索引不能提供高的可扩展性和可靠性，并且HDFS和HPC存储架构都具有诸如单点故障和不够高效的数据访问的缺点。该项目在以数据为中心的网络基础设施中构建可扩展的高可用性数据能力，以克服缺点，并使用分布式负载平衡，数据复制和一致性维护的新技术创建高度可扩展的文件系统。

This project supports data-related analysis in a wide range of science and engineering applications. It will contribute to the development of scalable data-centric cyberinfrastructure capabilities, to accelerate interdisciplinary and collaborative research. The exascale file system will serve as catalyst for research in data storage architectures, and will enable new data-focused services and capabilities that advance scientific discoveries, collaborations, and innovations. The project addresses a major data challenge common to a range of communities such as social science, economics, and bioengineering, and will provide thorough training and collaborative research opportunities for project participants. Both high performance computing (HPC) clusters and Hadoop clusters use file systems. A Hadoop cluster uses the Hadoop Distributed File System (HDFS) that resides on compute nodes, while an HPC cluster usually uses a remote storage system. Despite years of efforts on research and application development on HPC and Hadoop clusters, the file systems in both types of clusters still face a formidable challenge, that of achieving exascale computing capabilities. The centralized data indexing in HDFS and HPC storage architectures cannot provide high scalability and reliability, and both HDFS and HPC storage architectures have shortcomings such as single point of failure and insufficiently efficient data access. This project builds scalable high-availability data capabilities in data-centric cyberinfrastructure to overcome the shortcomings and create a highly scalable file system with new techniques for distributed load balancing, data replication and consistency maintenance.