Collaborative Research: Dynamic Staging Architecture for Accelerating I/O Pipelines

PB级应用程序正在以极高的速度产生TB级的数据。大型机器中的存储系统压力很大，因为I/O速率的增长速度不足以应对数据生产。各种HPC活动，如写入输出和检查点数据，都受到I/O带宽瓶颈的阻碍。此外，由于超级计算机和用户本地集群的存储/处理能力之间的差距越来越大，计算结果的后处理和随后的分析/可视化越来越耗时。本研究致力于构建一种新颖的在线动态数据转移体系结构，并将其应用于迫在眉睫的千万亿次I/O危机。为此，研究了以下目标：(I)协调使用节点本地存储器和来自专用节点集的新兴硬件(例如固态磁盘(SSD))，作为缓解I/O带宽瓶颈的手段；(Ii)在分级场地上利用异步I/O对传统用户后处理流水线和二级计算进行多路复用以执行可扩展的I/O和数据分析；(Iii)绕过存储器访问分级区域；以及(Iv)在分级场地以及将其连接到计算客户端和持久存储的通信通道中启用服务质量。这项研究将对未来极端规模机器的配置产生广泛的影响，并将为此提供形成性的指导方针。这项研究的结果将是一套可以从根本上改变当前并行I/O模型并加速PB级I/O流水线的集成技术。此外，这项研究将有助于分析SSD在日常超级计算I/O中的效用，并向更广泛的HPC社区告知其可行性。

Petascale applications are producing terabytes of data at a great rate. Storage systems in large-scale machines are significantly stressed as I/O rates are not growing as fast to cope with data production. A variety of HPC activities such as writing output and checkpoint data are all stymied by the I/O bandwidth bottleneck. Further to this, the post-processing and subsequent analysis/visualization of computational results is increasingly time consuming due to the widening gap between the storage/processing capacities of supercomputers and users' local clusters. This research focuses on building a novel in-job dynamic data staging architecture and in bringing it to bear on the looming petascale I/O crisis. To this end, the following objectives are investigated: (i) the concerted use of node-local memory and emerging hardware such as Solid State Disks (SSDs), from a dedicated set of nodes, as a means to alleviate the I/O bandwidth bottleneck, (ii) the multiplexing of traditional user post-processing pipelines and secondary computations with asynchronous I/O on the staging ground to perform scalable I/O and data analytics, (iii) bypassing memory to access the staging area, and (iv) enabling QoS both in the staging ground and in the communication channel connecting it to compute client and persistent storage. This study will have a wide-ranging impact on future provisioning of extreme-scale machines and will provide formative guidelines to this end. The result of this research will be a set of integrated techniques that can fundamentally change the current parallel I/O model and accelerate petascale I/O pipelines. Further, this research will help analyze the utility of SSDs in day-to-day supercomputing I/O and inform the wider HPC community of its viability.