Measuring the impact of burst buffers on data-intensive scientific workflows
Measuring the impact of burst buffers on data-intensive scientific workflows
复制标题
DOI：
10.1016/j.future.2019.06.016
复制
发表时间：
2019-12
期刊：
影响因子：
--
通讯作者：
Rafael Ferreira da Silva;S. Callaghan;T. Do;G. Papadimitriou;E. Deelman
中科院分区：
文献类型：
--
作者：
Rafael Ferreira da Silva;S. Callaghan;T. Do;G. Papadimitriou;E. Deelman
科学应用程序经常产生和消耗大量数据，但将这些数据传输到计算资源或从计算资源传输这些数据可能具有挑战性，因为并行文件系统的性能跟不上计算和内存的性能。为了缓解这一I/O瓶颈，一些系统已经部署了突发缓冲区，但它们对真实世界的科学工作流应用程序的性能的影响仍然不清楚。在本文中，我们研究了通过远程共享的、可分配的突发缓存对NERSC的CORI系统的影响。通过运行两个数据密集型工作流(高吞吐量基因组分析工作流)和SCEC高性能CyberShake工作流(生产地震风险分析工作流)的子集，我们发现使用猝发缓冲区可以提供数量级的读写改进，这些改进导致作业性能的提高，从而提高了整体工作流性能，即使是对长时间运行的CPU受限的作业也是如此。
Science applications frequently produce and consume large volumes of data, but delivering this data to and from compute resources can be challenging, as parallel file system performance is not keeping up with compute and memory performance. To mitigate this I/O bottleneck, some systems have deployed burst buffers, but their impact on performance for real-world scientific workflow applications is still not clear. In this paper, we examine the impact of burst buffers through the remote-shared, allocatable burst buffers on the Cori system at NERSC. By running two data-intensive workflows, a high-throughput genome analysis workflow, and a subset of the SCEC high-performance CyberShake workflow, a production seismic hazard analysis workflow, we find that using burst buffers offers read and write improvements of an order of magnitude, and these improvements lead to increased job performance, and thereby increased overall workflow performance, even for long-running CPU-bound jobs.