CAREER: A Scalable Hierarchical Framework for High-Performance Data Storage

现代科学应用，如分析来自大规模分布式传感器的信息、气候监测和环境影响预测，需要强大的计算资源，并需要管理不断增长的数据量。虽然由数万个或更多处理器组成的高端计算机体系结构正在成为支持此类应用的现代高性能计算(HPC)系统中的标准，但计算能力的这种增长并没有与存储和I/O系统的相应改进相匹配。因此，存储系统的性能和集群的计算能力之间的差距越来越大，这构成了严峻的挑战，特别是在支持新兴的千万亿级科学应用方面。这项研究开发了一个框架，用于弥合上述性能差距，并支持高性能计算机高效可靠的数据管理。通过框架的创新、设计、开发和部署，研究人员提高了现代高性能计算环境的I/O性能，目标高性能计算环境面临着独特的研究挑战，即通过增加存储容量来保持I/O性能，以低成本管理大量资源，进行大容量远程数据传输，以及适应应用程序不断变化的I/O需求。本研究通过为HPC数据存储采用可扩展的分层框架来解决存储管理中的这些挑战。该框架通过存储资源的分层组织、站点之间的分散交互以支持高速、大容量数据交换和战略性数据放置以及系统范围的I/O优化，在HPC群集站点内提供高性能可靠的存储。总体目标是建立一个符合现代HPC应用需求的数据存储框架，以缓解计算资源和I/O系统之间的潜在性能差距。本研究采用整体的方法，所有系统组件相互作用，以产生一个高效的高性能计算机数据管理系统。

Modern scientific applications, such as analyzing information from large-scale distributed sensors, climate monitoring, and forecasting environmental impacts, require powerful computing resources and entail managing an ever-growing amount of data. While high-end computer architectures comprising of tens-of-thousands or more processors are becoming a norm in modern High Performance Computing (HPC) systems supporting such applications, this growth in computational power has not been matched by a corresponding improvement in storage and I/O systems. Consequently, there is an increasing gap between storage system performance and computational power of clusters, which poses critical challenges, especially in supporting emerging petascale scientific applications. This research develops a framework for bridging the said performance gap and supporting efficient and reliable data management for HPC. Through innovation, design, development, and deployment of the framework, the investigators improve the I/O performance of modern HPC setups.The target HPC environments present unique research challenges, namely, maintaining I/O performance with increasing storage capacity, low-cost administration of a large number of resources, high-volume long-distance data transfers, and adapting to the varying I/O demands of applications. This research addresses these challenges in storage management by employing a Scalable Hierarchical Framework for HPC data storage. The framework provides high-performance reliable storage within HPC cluster sites via hierarchical organization of storage resources, decentralized interactions between sites to support high-speed, high-volume data exchange and strategic data placement, and system-wide I/O optimizations. The overall goal is a data storage framework attuned to the needs of modern HPC applications, which mitigates the underlying performance gap between compute resources and the I/O system. This research adopts a holistic approach where all system components interact to yield an efficient data management system for HPC.