VeloC: Towards High Performance Adaptive Asynchronous Checkpointing at Large Scale
VeloC: Towards High Performance Adaptive Asynchronous Checkpointing at Large Scale
复制标题
VeloC：迈向大规模高性能自适应异步检查点
DOI：
--
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
F. Cappello
中科院分区：
文献类型：
--
作者：
Bogdan Nicolae;A. Moody;Elsa Gonsiorowski;K. Mohror;F. Cappello
全局检查点指向外部存储（例如，并行文件系统）是许多HPC应用程序的常见I/O模式。但是，由于外部存储的I/O吞吐量有限，全局检查点通常会导致I/O瓶颈。为了解决这个问题，从同步检查点（即，阻塞直到完成写入）到异步检查点（即，写入到更快的本地存储器并在后台刷新到外部存储器）正被越来越多地采用。然而，随着每个节点的核心数量的增加以及本地和外部存储的异构性，由于在节点本地和全局级别上的高并发性和I/O性能可变性之间的复杂相互作用，设计高效的异步检查点机制并非易事。这个问题还没有得到很好的理解，但对现代超级计算基础设施非常重要。本文提出了一个通用的异步检查点解决方案，解决这个问题。为此，我们引入了一个并发优化的技术，结合性能建模与轻量级的监控，以作出明智的决定，使用什么本地存储设备，以动态适应后台刷新，并减少检查点开销。我们使用VeloC原型来说明这种技术。在前Exascale超级计算系统上进行的广泛实验显示出显着的好处。
Global checkpointing to external storage (e.g., a parallel file system) is a common I/O pattern of many HPC applications. However, given the limited I/O throughput of external storage, global checkpointing can often lead to I/O bottlenecks. To address this issue, a shift from synchronous checkpointing (i.e., blocking until writes have finished) to asynchronous checkpointing (i.e., writing to faster local storage and flushing to external storage in the background) is increasingly being adopted. However, with rising core count per node and heterogeneity of both local and external storage, it is non trivial to design efficient asynchronous checkpointing mechanisms due to the complex interplay between high concurrency and I/O performance variability at both the node-local and global levels. This problem is not well understood but highly important for modern supercomputing infrastructures. This paper proposes a versatile asynchronous checkpointing solution that addresses this problem. To this end, we introduce a concurrency-optimized technique that combines performance modeling with lightweight monitoring to make informed decisions about what local storage devices to use in order to dynamically adapt to background flushes and reduce the checkpointing overhead. We illustrate this technique using the VeloC prototype. Extensive experiments on a pre-Exascale supercomputing system show significant benefits.