Collaborative Research: SHF: Small: Scalable and Extensible I/O Runtime and Tools for Next Generation Adaptive Data Layouts

在超级计算机上进行大规模科学模拟的能力推动了一波跨越能源、宇宙学、地球科学、医学和国家安全等一系列学科的创新和发现。随着亿级的到来，应用程序承诺以更高的分辨率和保真度提供不断增长的大小数据。当前高性能计算(HPC)系统的技术趋势正在计算和I/O性能之间造成前所未有的差距，使数据移动成为模拟分析管道中最慢的组件。已经提出了许多技术来缓解这一瓶颈，包括压缩和分层数据布局，但当前的解决方案缺乏可扩展性和可移植性，并且没有为并行I/O和分析(就地和事后)工作流的数据管理需求提供整体方法。这项工作将为下一代自适应数据布局开发可伸缩和可扩展的I/O运行时和工具，内在地吸收压缩和渐进式数据访问，推动高性能数据管理领域的最先进水平。这项工作将为端到端数据管理解决方案奠定基础，该解决方案将满足整个模拟分析流水线的挑战性需求，并显著加速科学研究。该研究旨在为下一代自适应数据布局开发端到端数据管理解决方案。建议的数据布局将是分层的、压缩的和可调的，使其适合处理数据泛滥和HPC不断发展的格局。分层布局将允许逐步访问大规模数据，从而能够进行任何规模的事后和现场分析。最先进的数据压缩和缩减技术将显著缓解数据移动瓶颈，特别是在执行并行I/O时。最后，可调布局与新的性能分析和可视化工具相结合，将允许数据驱动的方法在运行时优化不同工作流和HPC平台的I/O性能。该项目旨在通过以下方式实现其目标：可扩展和可调的并行I/O运行时，其将支持使用自适应数据布局的渐进式读/写操作；接口，以支持用于现场工作流的自适应数据布局；新型WebGPU支持的可视化系统，其能够利用布局的渐进性，使得能够在Web浏览器上交互探索大数据集；以及性能挖掘和可视化工具，以实现数据驱动和可移植的I/O性能预测和自动调优。该解决方案将在Leadance超级计算机和中型集群上进行评估，并与大规模模拟、分析和I/O框架集成。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The ability to perform large scale scientific simulations on supercomputers have fueled a wave of innovation and discoveries across a range of disciplines including energy, cosmology, earth science, medicine, and national security. With the advent of exascale, applications promise to deliver data of ever-increasing size at higher resolution and fidelity. Current technology trends in High Performance Computing (HPC) systems are creating an unprecedented gap between compute and I/O performance, making data movement the slowest component of the simulation-analysis pipeline. Many techniques have been proposed to alleviate this bottleneck including compression and hierarchical data layouts, but current solutions lack scalability and portability, and do not provide a holistic approach to the data-management needs of both parallel I/O and analysis (in situ and post-hoc) workflows. This work will develop a scalable and extensible I/O runtime and tools for the next-generation adaptive data layouts that inherently imbibe compression and progressive data access, advancing the state of art in the field of high-performance data management. The work will lay the foundation for an end-to-end data management solution that will cater to the challenging needs of the entire simulation-analysis pipeline and significantly accelerate science at exascale.The research aims to develop an end-to-end data-management solution for the next generation adaptive data layouts. The proposed data layouts will be hierarchical, compressed, and tunable, making them suitable to deal with the data deluge and the evolving landscape of HPC. A hierarchical layout will allow progressive access to massively large data enabling post-hoc and in situ analysis at any scale. State-of-the-art data compression and reduction techniques will significantly alleviate data-movement bottlenecks, especially while performing parallel I/O. Finally, a tunable layout combined with novel performance analysis and visualization tools will allow data-driven approaches to optimize I/O performance at runtime for different workflows and HPC platforms. This project aims to achieve its goals by developing: a scalable and tunable parallel I/O runtime that will support progressive read/write operations using adaptive data layouts; interfaces to support the adaptive data layouts for in situ workflows; a novel WebGPU-powered visualization system that can take advantage of the progressive nature of the layout enabling interactive exploration of large datasets on web browsers; and performance-mining and -visualization tools to enable data-driven and portable I/O performance prediction and auto-tuning. The solution will be evaluated on leadership supercomputers and mid-scale clusters, and integrated with large-scale simulations, analysis, and I/O frameworks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.