SHF: Small: Collaborative Research: A Parallel Graph-Based Paradigm for HPC Parallel File System Checkers

现代高性能计算（HPC）平台依靠大规模并行文件系统为科学应用提供数据访问服务。这些并行文件系统通常运行在昂贵的硬件上，并且通常维护得很好，但是它们仍然可能由于各种原因（例如，硬件故障、软件错误、配置错误）而出现故障并进入不一致的状态。当状态变得不一致时，称为checker的检查和修复程序是使系统恢复一致性的最后一道防线。然而，今天的检查器容易出错，运行起来也很耗时。随着规模和复杂性的不断增加，情况可能会变得更糟。该项目旨在通过一种新的基于并行图的模型，实现广泛使用的并行文件系统的可扩展、高性能检查和修复。这个项目的成功将极大地改变并行文件系统检查器的使用方式。这种努力是为科学发现建立高度可靠的未来HPC并行文件系统的基本步骤。此外，该项目将研究活动与教育和外联工作相结合，以培养具有广泛包容性和全球竞争力的科学劳动力。该项目包括三个重点。第一个任务是构建一个通用的基于图的元数据模型来抽象关键元数据和一致性规则；第二个任务侧重于有效地从实际系统中检索元数据并实例化元数据图；第三个任务侧重于构建基于图的一致性检查运行时引擎，以并行执行检查，从而获得可伸缩的高性能。这包括构建用于表示不同文件系统元数据的通用图结构，提取用于检查的元数据项之间的一致性规则，以及定义一组接口以方便为其他文件系统构建图模型。该项目将探索将所有一致性规则编译成一个统一的可执行文件，称为blob？，可以在所有计算节点上并行运行，并优化运行时图引擎以适应依赖关系并实现高性能。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Modern high performance computing (HPC) platforms rely on large-scale parallel file systems for serving data accesses of scientific applications. These parallel file systems often run on expensive hardware and are usually well-maintained, but they may still experience failures and run into inconsistent states for various reasons (e.g., hardware faults, software bugs, configuration errors). When the state becomes inconsistent, a checking and repairing program called checker is the last line of defense to bring the system back to consistency. Nevertheless, today's checkers are error-prone and time-consuming to run. With the scale and complexity keeps increasing, the situation will likely get worse. This project aims to enable scalable, high performance checking and repairing of widely used parallel file systems through a new parallel graph-based model. The success of this project will dramatically change how parallel file system checkers would be used. Such an effort is a fundamental step towards building highly reliable future HPC parallel file systems for scientific discovery. In addition, this project integrates the research activities with education and outreach efforts to train broadly inclusive and globally competitive science workforce. The project consists of three thrusts. The first task focuses on constructing a general graph-based metadata model to abstract key metadata and consistency rules; the second task focuses on efficiently retrieving metadata from real systems and instantiating metadata graphs; the third task focuses on building a graph-based consistency checking runtime engine to conduct the checking in parallel to gain scalable high performance. This includes constructing a generic graph structure for representing different file system metadata, extracting the consistency rules among metadata items for checking, and defining a set of interfaces to facilitate building the graph model for other file systems. The project will explore compiling all consistency rules into a unified executable called ?blob?, which can be run in parallel in all compute nodes, and optimize the runtime graph engine to accommodate dependencies and achieve high performance.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.