CAREER: A Scalable Hierarchical Framework for High-Performance Data Storage

现代科学应用，如分析来自大规模分布式传感器的信息、气候监测和预测环境影响，需要强大的计算资源，并需要管理不断增长的数据量。虽然包括数万个或更多处理器的高端计算机架构正在成为支持这种应用的现代高性能计算（HPC）系统中的标准，但是计算能力的这种增长还没有与存储和I/O系统的相应改进相匹配。因此，存储系统性能和集群计算能力之间的差距越来越大，这带来了严峻的挑战，特别是在支持新兴的千万亿次科学应用方面。这项研究开发了一个框架，以弥合上述性能差距，并支持高效和可靠的数据管理HPC。通过框架的创新、设计、开发和部署，研究人员提高了现代HPC设置的I/O性能。目标HPC环境提出了独特的研究挑战，即在增加存储容量的情况下保持I/O性能、大量资源的低成本管理、大容量长距离数据传输以及适应应用程序的不同I/O需求。本研究通过采用HPC数据存储的可扩展分层框架来解决存储管理中的这些挑战。该框架通过存储资源的分层组织、站点之间的分散式交互以支持高速、大容量数据交换和战略数据放置以及系统范围的I/O优化，在HPC群集站点内提供高性能可靠的存储。总体目标是一个适应现代HPC应用需求的数据存储框架，它可以缓解计算资源和I/O系统之间的潜在性能差距。本研究采用了一种整体的方法，所有系统组件相互作用，产生一个高效的数据管理系统的HPC。

Modern scientific applications, such as analyzing information from large-scale distributed sensors, climate monitoring, and forecasting environmental impacts, require powerful computing resources and entail managing an ever-growing amount of data. While high-end computer architectures comprising of tens-of-thousands or more processors are becoming a norm in modern High Performance Computing (HPC) systems supporting such applications, this growth in computational power has not been matched by a corresponding improvement in storage and I/O systems. Consequently, there is an increasing gap between storage system performance and computational power of clusters, which poses critical challenges, especially in supporting emerging petascale scientific applications. This research develops a framework for bridging the said performance gap and supporting efficient and reliable data management for HPC. Through innovation, design, development, and deployment of the framework, the investigators improve the I/O performance of modern HPC setups.The target HPC environments present unique research challenges, namely, maintaining I/O performance with increasing storage capacity, low-cost administration of a large number of resources, high-volume long-distance data transfers, and adapting to the varying I/O demands of applications. This research addresses these challenges in storage management by employing a Scalable Hierarchical Framework for HPC data storage. The framework provides high-performance reliable storage within HPC cluster sites via hierarchical organization of storage resources, decentralized interactions between sites to support high-speed, high-volume data exchange and strategic data placement, and system-wide I/O optimizations. The overall goal is a data storage framework attuned to the needs of modern HPC applications, which mitigates the underlying performance gap between compute resources and the I/O system. This research adopts a holistic approach where all system components interact to yield an efficient data management system for HPC.