Framework: Software: NSCI: Collaborative Research: Hermes: Extending the HDF Library to Support Intelligent I/O Buffering for Deep Memory and Storage Hierarchy Systems

现代高性能计算(HPC)应用程序会产生海量数据。然而，基于磁盘的存储系统的性能改进一直比内存慢得多，这造成了显著的输入/输出(I/O)性能差距。为了缩小性能差距，存储子系统正在经历广泛的变化，采用新技术，并在内存/存储层次结构中添加更多层。随着更深层次的存储体系，存储系统的数据移动复杂性显著增加，使得更难利用深层存储和存储体系(DMSH)设计的潜力。随着我们迈向亿级时代，I/O瓶颈是解决HPC社区面临的性能瓶颈的必由之路。具有多层内存/存储层的DMSH提供了可行的解决方案，但要有效地使用却非常复杂。理想情况下，多层存储的存在对应用程序应该是透明的，而不必牺牲I/O性能。需要加强和扩展现有的软件系统，以支持在DMSH下透明和有效地访问和移动数据。分层数据格式(HDF)技术是一组当前的I/O解决方案，用于解决组织、访问、分析和保存数据方面的问题。HDF5库在科学界广受欢迎。在能源部实验室使用的高级I/O库中，HDF5是不可否认的领导者，拥有99%的份额。HDF5通过隐藏对单个共享文件执行协调I/O的复杂性，并通过封装通用优化来解决I/O瓶颈。尽管HDF技术与其他现有的I/O中间件一样，并不是为支持DMSH而设计的，但其广泛的受欢迎程度及其中间件特性使HDF5成为在DMSH下启用、管理和监督I/O缓冲的理想候选者。该项目建议开发Hermes，这是一个支持异构性、多层、动态和分布式的I/O缓冲系统，将显著提高I/O性能。该项目建议使用Hermes设计来扩展HDF技术。爱马仕是新的，HDF5的增强是新的。这项研究的成果包括一个增强的HDF5库，一套扩展的HDF技术，以及一组通用的I/O缓冲和内存系统优化机制和方法。我们相信，DMSH I/O缓冲和HDF技术的结合是一种可实现的实用解决方案，可以有效地支持科学发现。Hermes将通过开发新的缓冲算法和机制来推动HDF5核心技术，以支持1)DMSH中的垂直和水平缓冲：在这里，垂直意味着本地访问不同级别的数据，水平意味着跨远程计算节点分布/收集数据；2)通过HDF5的选择性缓冲：这里的选择性意味着某些内存层，例如NVMe，仅用于选定的数据；3)通过在线系统配置的动态缓冲：缓冲模式可以根据消息流量动态改变；4)通过强化学习的自适应缓冲：通过学习应用的访问模式，我们可以在运行时调整预取算法和缓存替换策略。开发的Hermes将被翻译成高质量可靠的软件，并将与核心HDF5库一起发布。该奖项反映了NSF的法定使命，并已通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Modern high performance computing (HPC) applications generate massive amounts of data. However, the performance improvement of disk based storage systems has been much slower than that of memory, creating a significant Input/Output (I/O) performance gap. To reduce the performance gap, storage subsystems are under extensive changes, adopting new technologies and adding more layers into the memory/storage hierarchy. With a deeper memory hierarchy, the data movement complexity of memory systems is increased significantly, making it harder to utilize the potential of the deep memory and storage hierarchy (DMSH) design. As we move towards the exascale era, I/O bottleneck is a must to solve performance bottleneck facing the HPC community. DMSHs with multiple levels of memory/storage layers offer a feasible solution but are very complex to use effectively. Ideally, the presence of multiple layers of storage should be transparent to applications without having to sacrifice I/O performance. There is a need to enhance and extend current software systems to support data access and movement transparently and effectively under DMSHs. Hierarchical Data Format (HDF) technologies are a set of current I/O solutions addressing the problems in organizing, accessing, analyzing, and preserving data. HDF5 library is widely popular within the scientific community. Among the high level I/O libraries used in DOE labs, HDF5 is the undeniable leader with 99% of the share. HDF5 addresses the I/O bottleneck by hiding the complexity of performing coordinated I/O to single, shared files, and by encapsulating general purpose optimizations. While HDF technologies, like other existing I/O middleware, are not designed to support DMSHs, its wide popularity and its middleware nature make HDF5 an ideal candidate to enable, manage, and supervise I/O buffering under DMSHs. This project proposes the development of Hermes, a heterogeneous aware, multi tiered, dynamic, and distributed I/O buffering system that will significantly accelerate I/O performance. This project proposes to extend HDF technologies with the Hermes design. Hermes is new, and the enhancement of HDF5 is new. The deliveries of this research include an enhanced HDF5 library, a set of extended HDF technologies, and a group of general I/O buffering and memory system optimization mechanisms and methods. We believe that the combination of DMSH I/O buffering and HDF technologies is a reachable practical solution that can efficiently support scientific discovery. Hermes will advance HDF5 core technology by developing new buffering algorithms and mechanisms to support 1) vertical and horizontal buffering in DMSHs: here vertical means access data to/from different levels locally and horizontal means spread/gather data across remote compute nodes; 2) selective buffering via HDF5: here selective means some memory layer, e.g. NVMe, only for selected data; 3) dynamic buffering via online system profiling: the buffering schema can be changed dynamically based on messaging traffic; 4) adaptive buffering via Reinforcement Learning: by learning the application's access pattern, we can adapt prefetching algorithms and cache replacement policies at runtime. The development Hermes will be translated into high quality dependable software and will be released with the core HDF5 library.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.