NSCI Elements: Software - PFSTRASE - A Parallel FileSystem TRacing and Analysis SErvice to Enhance Cyberinfrastructure Performance and Reliability

该项目将开发一个开源软件服务，并行文件系统跟踪和分析服务（PFSTRASE），提高国家数据存储系统的可靠性和性能。最大的超级计算机。 随着模拟和计算更忠实地代表现实，它们的规模沿着它们消耗和生成的数据的大小而迅速增长。 为了处理这些数据的存储和移动，超级计算系统建立在大规模并行数据存储系统的基础上。 由于它们的并行性质，这些存储系统能够以传统存储系统的数百倍的速度移动数据，从而实现否则不切实际的计算。 这些存储系统提供的性能能力伴随着复杂性，这导致它们的功能通常明显低于最佳状态，甚至在某些情况下失败。 这导致了计算时间的浪费，并最终失去了科学进步。 能够揭示这些问题并提高存储系统可靠性和性能的工具的开发状态对于当前和未来的计算系统是不充分的。 PFSTRASE将通过持续自动监控存储系统的健康和性能来填补这一空白，通过易于使用的界面提供见解，从而提高存储和超级计算机系统的可靠性和性能。并行文件系统（PFS）是高性能计算（HPC）架构中最关键的高可用性组件，为运行计算、用户和系统服务运行的环境以及应用程序和数据的存储提供输入/输出（I/O）服务。由于这一中心作用，PFS中的故障或性能下降事件会影响HPC资源的每个用户。PFS事件必须由系统管理员快速有效地处理;然而，通常没有足够的信息来建立PFS活动和事件之间的精确因果关系，这阻碍了及时和有针对性的补救措施的实施。为了填补这一信息空白，将开发一个开源的并行文件系统跟踪和分析服务（PFSTRASE），跟踪和分析必要的数据，以建立PFS活动与已实现和即将发生的事件之间的因果关系。该项目将为开源Lustre文件系统实现服务，这是大规模HPC站点最常用的PFS。特定PFS目录和文件操作的负载将被测量并合并到服务中，以构建来自每个作业、进程和用户的真实服务器负载贡献。仪式？的基础架构将持续监控整个PFS，并生成实时、无缝的表示，将作业、流程和用户的贡献与存储服务器负载、网络带宽和存储容量连接起来。该基础设施将提供一个易于导航的Web界面，以可视化格式呈现这些数据，包括实时和历史数据。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project will develop an open-source software service, the Parallel FileSystem TRacing and Analysis SErvice (PFSTRASE), that improves the reliability and performance of data storage systems for the nation?s largest supercomputers. As simulations and computations represent reality more faithfully they grow commensurately in scale along with the size of the data they consume and generate. To handle the storage and movement of this data, supercomputing systems are built on the backbone of massively parallel data storage systems. Due to their parallel nature these storage systems are capable of moving data at hundreds of times the speed of conventional storage systems, enabling otherwise impractical computations. The performance capabilities these storage systems provide is accompanied by a complexity that results in them often functioning significantly less than optimally and even in some instances failing. This results in wasted computational time and ultimately lost scientific progress. The state of development of tools that could cast light on these problems and improve storage system reliability and performance is inadequate for current and future computing systems. PFSTRASE will fill this gap by continually and automatically monitoring storage system health and performance, providing insights through an easy to use interface that will improve the reliability and performance of storage and supercomputer systems. Parallel filesystems (PFSs) are the most critical high-availability components of High Performance Computing (HPC) architectures, providing input/output (I/O) services to running computations, the environment that users and system services operate in, and storage for applications and data. Because of this central role, failure or performance degradation events in the PFS impact every user of an HPC resource. PFS events must be dealt with quickly and effectively by system administrators; however, there is typically insufficient information to establish precise causal relationships between PFS activity and events, impeding the implementation of timely and targeted remedies. To fill this information gap, an open-source Parallel FileSystem TRacing and Analysis SErvice (PFSTRASE) that traces and analyzes the requisite data to establish causal relationships between PFS activity and both realized and imminent events will be developed. This project will implement the service for the open-source Lustre filesystem, which is the most commonly used PFS at large-scale HPC sites. Loads for specific PFS directory and file operations will be measured and incorporated into the service to construct authentic server load contributions from every job, process, and user. The service?s infrastructure will continuously monitor the entire PFS and generate a real-time, seamless representation that connects contributions of jobs, processes, and users to storage server loads, network bandwidth, and storage capacities. The infrastructure will provide an easily navigable web interface that presents this data, both real-time and historical, in a visual format.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.