A Study of Failure Recovery and Logging of High-Performance Parallel File Systems
A Study of Failure Recovery and Logging of High-Performance Parallel File Systems
复制标题
DOI：
10.1145/3483447
复制
发表时间：
2022-03
期刊：
影响因子：
--
通讯作者：
Runzhou Han;Om Rameshwar Gatla;Mai Zheng;Jinrui Cao;Di Zhang;Dong Dai;Yong Chen;J. Cook
中科院分区：
文献类型：
--
作者：
Runzhou Han;Om Rameshwar Gatla;Mai Zheng;Jinrui Cao;Di Zhang;Dong Dai;Yong Chen;J. Cook
大规模并行文件系统（PFSS）在高性能计算（HPC）中起着至关重要的作用。但是，尽管它们的重要性与本地存储系统或云存储系统相比，其可靠性的研究或理解要少得多。实际HPC中心最近发生的故障事件已暴露了PFS簇中的潜在缺陷，并迫切需要进行系统分析。为了应对挑战，我们在本文中对PFSS的故障恢复和记录机制进行了研究。首先，为了触发目标PFS的故障恢复和记录操作，我们引入了一个名为PFAULT的黑盒故障注入工具，该工具对PFSS透明，易于在实践中部署。 PFAULT基于一组预定义的故障模型模拟了PFS中单个存储节点的故障状态，并可以系统地检查故障下的PFS行为。接下来，我们将PFAULT应用于研究两个广泛使用的PFS：光泽和BeEGFS。我们的分析揭示了目标PFSS的独特故障恢复和记录模式，并确定了多种情况，其中PFS在故障处理方面不完善。例如，光泽包括一个称为LFSCK的恢复组件来检测和修复PFS级别的不一致，但是我们发现，当扫描损坏的光泽时，LFSCK本身可能会悬挂或触发内核。即使在LFSCK的恢复尝试之后，应用于光泽的后续工作负载仍然异常行为（例如，挂起或报告I/O错误）。在BeEGF及其恢复组件BeeGFS-FSCK中也观察到了类似的问题。我们分析了深度观察到的异常症状的根本原因，这导致将新的斑块合并到即将到来的光泽释放中。此外，我们详细介绍了实验中生成的广泛日志，并在故障日志记录方面确定了PFSS的唯一模式和局限性。我们希望这项研究以及生成的工具和数据集可以促进社区的后续研究，并有助于改善PFSS以获得可靠的高性能计算。
Large-scale parallel file systems (PFSs) play an essential role in high-performance computing (HPC). However, despite their importance, their reliability is much less studied or understood compared with that of local storage systems or cloud storage systems. Recent failure incidents at real HPC centers have exposed the latent defects in PFS clusters as well as the urgent need for a systematic analysis. To address the challenge, we perform a study of the failure recovery and logging mechanisms of PFSs in this article. First, to trigger the failure recovery and logging operations of the target PFS, we introduce a black-box fault injection tool called PFault, which is transparent to PFSs and easy to deploy in practice. PFault emulates the failure state of individual storage nodes in the PFS based on a set of pre-defined fault models and enables examining the PFS behavior under fault systematically. Next, we apply PFault to study two widely used PFSs: Lustre and BeeGFS. Our analysis reveals the unique failure recovery and logging patterns of the target PFSs and identifies multiple cases where the PFSs are imperfect in terms of failure handling. For example, Lustre includes a recovery component called LFSCK to detect and fix PFS-level inconsistencies, but we find that LFSCK itself may hang or trigger kernel panics when scanning a corrupted Lustre. Even after the recovery attempt of LFSCK, the subsequent workloads applied to Lustre may still behave abnormally (e.g., hang or report I/O errors). Similar issues have also been observed in BeeGFS and its recovery component BeeGFS-FSCK. We analyze the root causes of the abnormal symptoms observed in depth, which has led to a new patch set to be merged into the coming Lustre release. In addition, we characterize the extensive logs generated in the experiments in detail and identify the unique patterns and limitations of PFSs in terms of failure logging. We hope this study and the resulting tool and dataset can facilitate follow-up research in the communities and help improve PFSs for reliable high-performance computing.