Asynchronous Receiver-Driven Replay for Local Rollback of MPI Applications
Asynchronous Receiver-Driven Replay for Local Rollback of MPI Applications
复制标题
DOI：
10.1109/ftxs49593.2019.00006
复制
发表时间：
2019-11
期刊：
影响因子：
--
通讯作者：
Nuria Losada;A. Bouteiller;G. Bosilca
中科院分区：
文献类型：
--
作者：
Nuria Losada;A. Bouteiller;G. Bosilca
随着超级计算机的规模和架构复杂性的增加，故障管理已经成为成功执行长时间运行的高性能计算应用程序的组成部分。在许多情况下，故障具有局部范围，通常影响正在使用的资源的子集，但广泛使用的故障恢复策略（如检查点/重启）无法利用并依赖于全局同步恢复操作。即使使用本地回滚恢复，其中只有故障影响的进程从检查点重新启动，执行中的进一步进展的一致性也通过从消息日志重放通信来实现。这种理论上合理的方法遇到了一些实际的限制：集体操作的存在迫使同步恢复，防止幸存者进程继续执行，消除了任何可能性重叠的进一步计算与恢复;恢复对等点所需的资源量可能是不可持续的。在这项工作中，我们解决了这两个问题，通过实现一个异步的，接收器驱动的点对点和集体通信的重放，并利用远程内存访问功能来访问消息日志。这个新的协议进行评估，在实现本地回滚的用户级故障缓解容错消息传递接口（MPI）。它将失败进程的恢复时间平均减少了59%，而幸存者进程在恢复中花费的时间与等效的全局回滚协议相比减少了95%，从而实现了恢复操作真正本地化影响的承诺。
With the increase in scale and architectural complexity of supercomputers, the management of failures has become integral to successfully executing a long-running high-performance computing application. In many instances, failures have a localized scope, usually impacting a subset of the resources being used, yet widely used failure recovery strategies (like checkpoint/restart) fail to take advantage and rely on global, synchronous recovery actions. Even with local rollback recovery, in which only the fault impacted processes are restarted from a checkpoint, the consistency of further progress in the execution is achieved through the replay of communication from a message log. This theoretically sound approach encounters some practical limitations: the presence of collective operations forces a synchronous recovery that prevents survivor processes from continuing their execution, removing any possibility for overlapping further computation with the recovery; and the amount of resources required at recovering peers can be untenable. In this work, we solved both problems by implementing an asynchronous, receiver-driven replay of point-to-point and collective communications, and by exploiting remote-memory access capabilities to access the message logs. This new protocol is evaluated in an implementation of local rollback over the User Level Failure Mitigation fault tolerant Message Passing Interface (MPI). It reduces the recovery times of the failed processes by an average of 59%, while the time spent in the recovery by the survivor processes is reduced by 95% when compared to an equivalent global rollback protocol, thus living to the promise of a truly localized impact of recovery actions.