Framework: Software: NSCI: Collaborative Research: Hermes: Extending the HDF Library to Support Intelligent I/O Buffering for Deep Memory and Storage Hierarchy Systems

现代高性能计算（HPC）应用程序生成大量数据。然而，基于磁盘的存储系统的性能改进比存储器的性能改进慢得多，从而产生了显著的输入/输出（I/O）性能差距。为了缩小性能差距，存储子系统正在进行广泛的更改，采用新技术并在内存/存储层次结构中添加更多层。随着更深的存储器层次结构，存储器系统的数据移动复杂性显著增加，使得更难利用深度存储器和存储层次结构（DMSH）设计的潜力。随着我们迈向亿级时代，I/O瓶颈是HPC社区必须解决的性能瓶颈。具有多级存储器/存储层的DMSH提供了一种可行的解决方案，但要有效使用非常复杂。理想情况下，多层存储的存在应该对应用程序透明，而不必牺牲I/O性能。有必要加强和扩大目前的软件系统，以支持在灾害管理和社会保障系统下透明和有效地获取和移动数据。分层数据格式（HDF）技术是一组当前的I/O解决方案，用于解决组织、访问、分析和保存数据方面的问题。HDF 5库在科学界广受欢迎。在DOE实验室使用的高级I/O库中，HDF 5是不可否认的领导者，拥有99%的份额。HDF 5通过隐藏对单个共享文件执行协调I/O的复杂性以及封装通用优化来解决I/O瓶颈。虽然HDF技术与其他现有的I/O中间件一样，并不是为支持DMSH而设计的，但其广泛的普及性及其中间件性质使HDF 5成为在DMSH下启用、管理和监督I/O缓冲的理想候选者。该项目提出了爱马仕，异构感知，多层，动态和分布式I/O缓冲系统，将显着加快I/O性能的发展。该项目建议采用爱马仕设计扩展HDF技术。爱马仕是新的，HDF 5的增强是新的。本研究的成果包括一个增强的HDF 5库，一组扩展的HDF技术，以及一组通用的I/O缓冲和存储系统优化机制和方法。我们相信，DMSH I/O缓冲和HDF技术的结合是一种可实现的实用解决方案，可以有效地支持科学发现。爱马仕将通过开发新的缓冲算法和机制来推进HDF 5核心技术，以支持1）DMSH中的垂直和水平缓冲：这里垂直意味着本地访问到/来自不同级别的数据，水平意味着跨远程计算节点传播/收集数据; 2）通过HDF 5进行选择性缓冲：这里选择性意味着仅用于选定数据的某个存储器层，例如NVMe; 3）通过在线系统分析的动态缓冲：可以根据消息流量动态改变缓冲模式; 4）通过强化学习的自适应缓冲：通过学习应用程序的访问模式，我们可以在运行时自适应预取算法和缓存替换策略。开发的爱马仕将被转化为高质量的可靠软件，并将与核心HDF 5库一起发布。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Modern high performance computing (HPC) applications generate massive amounts of data. However, the performance improvement of disk based storage systems has been much slower than that of memory, creating a significant Input/Output (I/O) performance gap. To reduce the performance gap, storage subsystems are under extensive changes, adopting new technologies and adding more layers into the memory/storage hierarchy. With a deeper memory hierarchy, the data movement complexity of memory systems is increased significantly, making it harder to utilize the potential of the deep memory and storage hierarchy (DMSH) design. As we move towards the exascale era, I/O bottleneck is a must to solve performance bottleneck facing the HPC community. DMSHs with multiple levels of memory/storage layers offer a feasible solution but are very complex to use effectively. Ideally, the presence of multiple layers of storage should be transparent to applications without having to sacrifice I/O performance. There is a need to enhance and extend current software systems to support data access and movement transparently and effectively under DMSHs. Hierarchical Data Format (HDF) technologies are a set of current I/O solutions addressing the problems in organizing, accessing, analyzing, and preserving data. HDF5 library is widely popular within the scientific community. Among the high level I/O libraries used in DOE labs, HDF5 is the undeniable leader with 99% of the share. HDF5 addresses the I/O bottleneck by hiding the complexity of performing coordinated I/O to single, shared files, and by encapsulating general purpose optimizations. While HDF technologies, like other existing I/O middleware, are not designed to support DMSHs, its wide popularity and its middleware nature make HDF5 an ideal candidate to enable, manage, and supervise I/O buffering under DMSHs. This project proposes the development of Hermes, a heterogeneous aware, multi tiered, dynamic, and distributed I/O buffering system that will significantly accelerate I/O performance. This project proposes to extend HDF technologies with the Hermes design. Hermes is new, and the enhancement of HDF5 is new. The deliveries of this research include an enhanced HDF5 library, a set of extended HDF technologies, and a group of general I/O buffering and memory system optimization mechanisms and methods. We believe that the combination of DMSH I/O buffering and HDF technologies is a reachable practical solution that can efficiently support scientific discovery. Hermes will advance HDF5 core technology by developing new buffering algorithms and mechanisms to support 1) vertical and horizontal buffering in DMSHs: here vertical means access data to/from different levels locally and horizontal means spread/gather data across remote compute nodes; 2) selective buffering via HDF5: here selective means some memory layer, e.g. NVMe, only for selected data; 3) dynamic buffering via online system profiling: the buffering schema can be changed dynamically based on messaging traffic; 4) adaptive buffering via Reinforcement Learning: by learning the application's access pattern, we can adaptprefetching algorithms and cache replacement policies at runtime. The development Hermes will be translated into high quality dependable software and will be released with the core HDF5 library.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.