CRFS: A Lightweight User-Level Filesystem for Generic Checkpoint/Restart
CRFS: A Lightweight User-Level Filesystem for Generic Checkpoint/Restart
复制标题
CRFS：用于通用检查点/重启的轻量级用户级文件系统
DOI：
--
复制
发表时间：
2011
期刊：
影响因子：
--
通讯作者：
D. Panda
中科院分区：
文献类型：
--
作者：
Xiangyong Ouyang;R. Rajachandrasekar;Xavier Besseron;Hao Wang;Jian Huang;D. Panda
检查点/重启（C/R）机制已被许多 MPI 库广泛采用 [1 - 3] 以实现容错。然而，这种机制的一个主要限制是由于需要将所有进程的快照转储到持久存储中而导致的密集IO瓶颈。已经进行了多项研究来最大限度地减少这种开销 [4, 5]，但大多数建议的优化都是在特定 MPI 堆栈或检查点库或应用程序内执行的，因此它们的可移植性不足以应用于其他 MPI 堆栈和应用程序。在本文中，我们提出了一种基于文件系统的方法来缓解这种检查点 IO 瓶颈。我们提出了一种新的文件系统，名为检查点重启文件系统（CRFS），它是一个基于 FUSE（用户空间文件系统）的轻量级用户级文件系统。 CRFS 的设计考虑了检查点/重启 I/O 流量，以有效处理并发写入请求。任何使用标准文件系统接口的软件组件都可以明显地从 CRFS 的功能中受益。 CRFS 拦截检查点文件写入系统调用，并将它们聚合成更少的更大块，这些块异步写入底层文件系统以提高 IO 效率。 CRFS 管理一个灵活的内部 IO 线程池来限制并发 IO，从而减轻 IO 争用，从而获得更好的 IO 性能。 CRFS 可以安装在任何标准文件系统上，例如 ext3、NFS 和 Lustre。我们已经实现了 CRFS 并使用三种流行的支持 C/R 的 MPI 堆栈评估了其性能：MVAPICH2、MPICH2 和 OpenMPI。实验结果显示所有三个 MPI 堆栈的性能都有显着提升。 CRFS 将检查点写入 Lustre 文件系统的性能提高了 5.5 倍。 ext3 和 NFS 文件系统也获得了类似水平的改进。据我们所知，这是第一个专为通用检查点/重启数据而设计的便携式轻量级文件系统。
Checkpoint/Restart (C/R) mechanisms have been widely adopted by many MPI libraries [1 -- 3] to achieve fault-tolerance. However, a major limitation of such mechanisms is the intensive IO bottleneck caused by the need to dump the snapshots of all processes into persistent storage. Several studies have been conducted to minimize this overhead [4, 5], but most of these proposed optimizations are performed inside specific MPI stack or check pointing library or applications, hence they are not portable enough to be applied to other MPI stacks and applications. In this paper, we propose a filesystem based approach to alleviate this checkpoint IO bottleneck. We propose a new filesystem, named Checkpoint-Restart File system (CRFS), which is a lightweight user-level filesystem based on FUSE (File system in User space). CRFS is designed with Checkpoint/Restart I/O traffic in mind to efficiently handle the concurrent write requests. Any software component using standard filesystem interfaces can transparently benefit from CRFS's capabilities. CRFS intercepts the checkpoint file write system calls and aggregates them into fewer bigger chunks which are asynchronously written to the underlying filesystem for more efficient IO. CRFS manages a ?exible internal IO thread pool to throttle concurrent IO to alleviate IO contention for better IO performance. CRFS can be mounted over any standard filesystem like ext3, NFS and Lustre. We have implemented CRFS and evaluated its performance using three popular C/R capable MPI stacks: MVAPICH2, MPICH2 and OpenMPI. Experimental results show significant performance gains for all three MPI stacks. CRFS achieves up to 5.5X speedup in checkpoint writing performance to Lustre filesystem. Similar level of improvements are also obtained with ext3 and NFS filesystems. To the best of our knowledge, this is the first such portable and light-weight filesystem designed for generic Checkpoint/Restart data.