SHF: Small: Collaborative Research: Uncovering Vulnerabilities in Parallel File Systems for Reliable High Performance Computing

许多科学问题（例如，计算生物学，高能物理，气候科学）依赖于高性能计算（HPC）系统来管理和处理大量数据。然而，随着规模和复杂性的快速增长，即使是专门设计和维护良好的HPC平台也可能失败。本研究旨在设计创新的方法，仔细检查并行文件系统（高性能计算平台的主要存储软件），并揭示并行文件系统在各种故障情况下可能导致数据丢失的问题。这种努力是构建高可靠的高性能计算系统和满足数据驱动的科学发现需求的基本步骤。此外，该项目将研究活动与教育和外联工作相结合，以培养具有广泛包容性和全球竞争力的科学劳动力。更具体地说，这个项目包括两个协同的研究任务，它支持自动测试和诊断并行文件系统中的问题。第一个任务侧重于通过单故障注入框架测试并行文件系统，该框架自动中断目标并行文件系统的正常工作负载，并检查中断是否会导致并行文件系统的相应检查器无法修复的任何问题。在第一个任务的基础上，第二个任务侧重于通过基于来源的两级分析来诊断前一个任务中发现的问题。第一级分析构建粗粒度的节点间溯源，它提供了整个系统行为的高级图像。第二级分析创建细粒度的节点内溯源，其中包含每个单独节点内的因果路径。此外，多个来源跟踪会自动对齐和比较，以帮助以最少的人力定位有问题的代码区域。

Many scientific problems (e.g., computational biology, high-energy physics, climate science) rely on high performance computing (HPC) systems to manage and process massive amounts of data. However, with the rapid increase in scale and complexity, even the specially-designed and well-maintained HPC platforms may fail. This research aims to design innovative methodologies that scrutinize parallel file systems, the major storage software which empowers HPC platforms, and uncover the issues in parallel file systems that can lead to data loss under various failure scenarios. Such an effort is a fundamental step towards building highly reliable HPC systems and meet the demand of data-driven scientific discovery. In addition, this project integrates the research activities with education and outreach efforts to train broadly inclusive and globally competitive science workforce.More specifically, this project includes two synergistic research tasks, which enables automatic testing as well as diagnosing the issues in parallel file systems. The first task focuses on testing parallel file systems through a single-fault injection framework, which interrupts the normal workloads of the target parallel file system automatically, and examines if the interruption could lead to any issues that cannot be fixed by the corresponding checker of the parallel file system. Building on the first task, the second task focuses on diagnosing the issues uncovered in the previous task through a two-level provenance-based analysis. The first level analysis builds the coarse-grain, inter-node provenance, which provides a high-level picture of the entire system behavior. The second level analysis creates the fine-grain, intra-node provenance that contains causal paths within each individual node. In addition, multiple provenance traces are aligned and compared automatically to help locate the problematic code region with minimal human efforts.