SHF: Medium: Collaborative Research: ECC: Ephemeral Coherence Cohort for I/O Containerization and Disaggregation

领先的高性能计算(HPC)计算设施在文件和存储系统方面投入了巨额资金。原因是HPC存储系统通常是HPC系统的阿喀琉斯之踵，因为它充满了争用、拥塞和性能变化无常的情况。这个问题变得越来越严重，原因是：(A)数据驱动的高性能计算的重要性增加，以及大规模模拟产生的数据量增加；以及(B)与CPU速度相比，磁盘速度的增长速度较慢。高带宽持久存储设备作为突发缓冲区的添加为快速缓存应用程序数据带来了新的机会，同时仍允许数据持久性。然而，将突发缓冲器用作又一个缓存层的传统方法不能减少在深层I/O堆栈中冗长且昂贵的数据处理步骤，也不能调和复杂存储系统内偶尔发生的争用。因此，该项目寻求利用突发缓冲区作为持久应用程序特定的并行文件系统的存储库，其生命周期与应用程序或HPC系统上的应用程序活动的生命周期相称。这是伊利诺伊大学香槟分校和佛罗里达州立大学的合作项目。该项目制定了一个称为临时一致性队列(ECC)的研究框架，该框架提供了一个抽象概念来表示通过集装化的应用程序数据的主动收集，隔离跨不同应用程序的I/O活动，并支持针对临时分配和突发缓冲区的动态利用的存储分解。拟议的ECC框架旨在增强能源部和国家科学基金会领导计算设施上运行的各种关键任务应用程序。该项目加强了伊利诺伊大学厄巴纳-香槟分校与佛罗里达州立大学的合作。该项目计划在即将到来的HPC会议上组织关于突发缓冲研究的小组和羽毛鸟会议，并与超级计算中心的领导人合作，利用这项研究的技术更广泛地渗透到社区。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Leadership computing facilities for high-performance computing (HPC) have a huge investment in the file and storage systems. The reason is that the HPC storage system often is the Achilles Heel of HPC systems, as it is fraught with numerous scenarios for contention, congestion and performance variability. This problem is getting worse due to: (a) the increased importance of data-driven HPC and the growth in the amount of data generated by large-scale simulation; and (b) the slower growth of disk speed, as compared to CPU speed. The addition of high-bandwidth persistent memory devices as burst-buffers brings in new opportunities for fast caching of application data while still allowing data persistence. However, the conventional approach of exploiting burst-buffers as yet another caching layer cannot reduce the lengthy and costly data processing steps in the deep I/O stack or reconcile occasional contentions inside the complex storage system. This project, therefore, seeks to exploit burst-buffers as repositories of persistent application-specific parallel file systems, with a lifetime commensurate to the lifetime of an application or an application campaign on a HPC system. This is a collaborative project between University of Illinois at Urbana-Champaign and Florida State University. This project formulates a research framework called Ephemeral Coherence Cohort (ECC) that offers an abstraction to represent the active collection of application data through containerization, insulate I/O activities across different applications, and enable storage disaggregation for ephemeral allocation and dynamic utilization of burst buffers. The proposed ECC framework aims to enhance a variety of mission-critical applications running on the Department of Energy and the National Science Foundation leadership computing facilities. The project strengthens the collaboration between University of Illinois Urbana-Champaign and the Florida State University. The project has plans to organize panels and birds-of-feather sessions on burst buffer research in the upcoming HPC conferences and collaborate with leaders of super-computing centers for wider community penetration with techniques from this research.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.