SHF: Small: Collaborative Research: Uncovering Vulnerabilities in Parallel File Systems for Reliable High Performance Computing

许多科学问题（例如，计算生物学、高能物理学、气候科学）依赖于高性能计算（HPC）系统来管理和处理大量数据。然而，随着规模和复杂性的快速增加，即使是专门设计和维护良好的HPC平台也可能失败。这项研究旨在设计创新的方法，审查并行文件系统，主要的存储软件，使HPC平台，并发现并行文件系统中的问题，可能导致数据丢失的各种故障情况下。这样的努力是构建高度可靠的HPC系统和满足数据驱动的科学发现需求的基本步骤。此外，该项目将研究活动与教育和推广工作相结合，以培养具有广泛包容性和全球竞争力的科学人才。更具体地说，该项目包括两个协同研究任务，这使得能够自动测试以及诊断并行文件系统中的问题。第一个任务是通过一个单故障注入框架对并行文件系统进行测试，该框架自动中断目标并行文件系统的正常工作负载，并检查中断是否会导致并行文件系统的相应检查器无法修复的任何问题。在第一项任务的基础上，第二项任务侧重于通过基于出处的两级分析来诊断前一项任务中发现的问题。第一级分析构建粗粒度的节点间起源，它提供了整个系统行为的高级描述。第二级分析创建了细粒度的节点内起源，其中包含每个节点内的因果路径。此外，多个起源跟踪会自动对齐和比较，以帮助以最少的人力定位有问题的代码区域。

Many scientific problems (e.g., computational biology, high-energy physics, climate science) rely on high performance computing (HPC) systems to manage and process massive amounts of data. However, with the rapid increase in scale and complexity, even the specially-designed and well-maintained HPC platforms may fail. This research aims to design innovative methodologies that scrutinize parallel file systems, the major storage software which empowers HPC platforms, and uncover the issues in parallel file systems that can lead to data loss under various failure scenarios. Such an effort is a fundamental step towards building highly reliable HPC systems and meet the demand of data-driven scientific discovery. In addition, this project integrates the research activities with education and outreach efforts to train broadly inclusive and globally competitive science workforce.More specifically, this project includes two synergistic research tasks, which enables automatic testing as well as diagnosing the issues in parallel file systems. The first task focuses on testing parallel file systems through a single-fault injection framework, which interrupts the normal workloads of the target parallel file system automatically, and examines if the interruption could lead to any issues that cannot be fixed by the corresponding checker of the parallel file system. Building on the first task, the second task focuses on diagnosing the issues uncovered in the previous task through a two-level provenance-based analysis. The first level analysis builds the coarse-grain, inter-node provenance, which provides a high-level picture of the entire system behavior. The second level analysis creates the fine-grain, intra-node provenance that contains causal paths within each individual node. In addition, multiple provenance traces are aligned and compared automatically to help locate the problematic code region with minimal human efforts.