Collaborative Research: SHF: Small: Scalable and Extensible I/O Runtime and Tools for Next Generation Adaptive Data Layouts

点击翻译按钮获取中文摘要

The ability to perform large scale scientific simulations on supercomputers have fueled a wave of innovation and discoveries across a range of disciplines including energy, cosmology, earth science, medicine, and national security. With the advent of exascale, applications promise to deliver data of ever-increasing size at higher resolution and fidelity. Current technology trends in High Performance Computing (HPC) systems are creating an unprecedented gap between compute and I/O performance, making data movement the slowest component of the simulation-analysis pipeline. Many techniques have been proposed to alleviate this bottleneck including compression and hierarchical data layouts, but current solutions lack scalability and portability, and do not provide a holistic approach to the data-management needs of both parallel I/O and analysis (in situ and post-hoc) workflows. This work will develop a scalable and extensible I/O runtime and tools for the next-generation adaptive data layouts that inherently imbibe compression and progressive data access, advancing the state of art in the field of high-performance data management. The work will lay the foundation for an end-to-end data management solution that will cater to the challenging needs of the entire simulation-analysis pipeline and significantly accelerate science at exascale.The research aims to develop an end-to-end data-management solution for the next generation adaptive data layouts. The proposed data layouts will be hierarchical, compressed, and tunable, making them suitable to deal with the data deluge and the evolving landscape of HPC. A hierarchical layout will allow progressive access to massively large data enabling post-hoc and in situ analysis at any scale. State-of-the-art data compression and reduction techniques will significantly alleviate data-movement bottlenecks, especially while performing parallel I/O. Finally, a tunable layout combined with novel performance analysis and visualization tools will allow data-driven approaches to optimize I/O performance at runtime for different workflows and HPC platforms. This project aims to achieve its goals by developing: a scalable and tunable parallel I/O runtime that will support progressive read/write operations using adaptive data layouts; interfaces to support the adaptive data layouts for in situ workflows; a novel WebGPU-powered visualization system that can take advantage of the progressive nature of the layout enabling interactive exploration of large datasets on web browsers; and performance-mining and -visualization tools to enable data-driven and portable I/O performance prediction and auto-tuning. The solution will be evaluated on leadership supercomputers and mid-scale clusters, and integrated with large-scale simulations, analysis, and I/O frameworks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.