Collaborative Research: SHF: Small: Scalable and Extensible I/O Runtime and Tools for Next Generation Adaptive Data Layouts

在超级计算机上进行大规模科学模拟的能力推动了一系列学科的创新和发现，包括能源，宇宙学，地球科学，医学和国家安全。随着兆兆级的出现，应用程序承诺以更高的分辨率和保真度提供不断增加的数据。高性能计算（HPC）系统的当前技术趋势正在计算和I/O性能之间产生前所未有的差距，使数据移动成为模拟分析管道中最慢的组件。已经提出了许多技术来缓解这一瓶颈，包括压缩和分层数据布局，但目前的解决方案缺乏可扩展性和可移植性，并没有提供一个整体的方法来满足并行I/O和分析（原位和事后）工作流的数据管理需求。这项工作将为下一代自适应数据布局开发一个可扩展和可扩展的I/O运行时和工具，这些自适应数据布局本质上吸收了压缩和渐进式数据访问，推进了高性能数据管理领域的最新技术。这项工作将为端到端的数据管理解决方案奠定基础，该解决方案将满足整个模拟分析管道的挑战性需求，并显着加快科学在exascale。该研究旨在为下一代自适应数据布局开发端到端的数据管理解决方案。拟议的数据布局将是分层的，压缩的和可调的，使它们适合处理数据泛滥和HPC不断发展的景观。分层布局将允许逐步访问海量数据，从而能够进行任何规模的事后和现场分析。最先进的数据压缩和精简技术将显著缓解数据移动瓶颈，特别是在执行并行I/O时。最后，可调布局与新颖的性能分析和可视化工具相结合，将允许数据驱动的方法在运行时优化不同工作流和HPC平台的I/O性能。该项目旨在通过开发以下各项来实现其目标：一个可伸缩和可调的并行I/O运行时，将支持使用自适应数据布局的渐进式读/写操作;支持现场工作流程的自适应数据布局的接口;一个新颖的WebGPU驱动的可视化系统，可以利用布局的渐进性，在Web浏览器上交互式探索大型数据集;以及性能挖掘和可视化工具，以实现数据驱动和便携式I/O性能预测和自动调优。该解决方案将在领先的超级计算机和中型集群上进行评估，并与大规模模拟，分析和I/O框架集成。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

The ability to perform large scale scientific simulations on supercomputers have fueled a wave of innovation and discoveries across a range of disciplines including energy, cosmology, earth science, medicine, and national security. With the advent of exascale, applications promise to deliver data of ever-increasing size at higher resolution and fidelity. Current technology trends in High Performance Computing (HPC) systems are creating an unprecedented gap between compute and I/O performance, making data movement the slowest component of the simulation-analysis pipeline. Many techniques have been proposed to alleviate this bottleneck including compression and hierarchical data layouts, but current solutions lack scalability and portability, and do not provide a holistic approach to the data-management needs of both parallel I/O and analysis (in situ and post-hoc) workflows. This work will develop a scalable and extensible I/O runtime and tools for the next-generation adaptive data layouts that inherently imbibe compression and progressive data access, advancing the state of art in the field of high-performance data management. The work will lay the foundation for an end-to-end data management solution that will cater to the challenging needs of the entire simulation-analysis pipeline and significantly accelerate science at exascale.The research aims to develop an end-to-end data-management solution for the next generation adaptive data layouts. The proposed data layouts will be hierarchical, compressed, and tunable, making them suitable to deal with the data deluge and the evolving landscape of HPC. A hierarchical layout will allow progressive access to massively large data enabling post-hoc and in situ analysis at any scale. State-of-the-art data compression and reduction techniques will significantly alleviate data-movement bottlenecks, especially while performing parallel I/O. Finally, a tunable layout combined with novel performance analysis and visualization tools will allow data-driven approaches to optimize I/O performance at runtime for different workflows and HPC platforms. This project aims to achieve its goals by developing: a scalable and tunable parallel I/O runtime that will support progressive read/write operations using adaptive data layouts; interfaces to support the adaptive data layouts for in situ workflows; a novel WebGPU-powered visualization system that can take advantage of the progressive nature of the layout enabling interactive exploration of large datasets on web browsers; and performance-mining and -visualization tools to enable data-driven and portable I/O performance prediction and auto-tuning. The solution will be evaluated on leadership supercomputers and mid-scale clusters, and integrated with large-scale simulations, analysis, and I/O frameworks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.