NSCI Elements: Software - PFSTRASE - A Parallel FileSystem TRacing and Analysis SErvice to Enhance Cyberinfrastructure Performance and Reliability

NSCI Elements：软件 - PFSTRASE - 用于增强网络基础设施性能和可靠性的并行文件系统跟踪和分析服务

• 批准号: 1835135
• 负责人: Richard Evans
• 金额: $ 38.59万
• 依托单位: University of Texas at Austin
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-10-01 至 2022-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1835135&HistoricalAwards=false
• 关键词: NSCI; Elements; Software; PFSTRASE; Parallel

This project will develop an open-source software service, the Parallel FileSystem TRacing and Analysis SErvice (PFSTRASE), that improves the reliability and performance of data storage systems for the nation?s largest supercomputers. As simulations and computations represent reality more faithfully they grow commensurately in scale along with the size of the data they consume and generate. To handle the storage and movement of this data, supercomputing systems are built on the backbone of massively parallel data storage systems. Due to their parallel nature these storage systems are capable of moving data at hundreds of times the speed of conventional storage systems, enabling otherwise impractical computations. The performance capabilities these storage systems provide is accompanied by a complexity that results in them often functioning significantly less than optimally and even in some instances failing. This results in wasted computational time and ultimately lost scientific progress. The state of development of tools that could cast light on these problems and improve storage system reliability and performance is inadequate for current and future computing systems. PFSTRASE will fill this gap by continually and automatically monitoring storage system health and performance, providing insights through an easy to use interface that will improve the reliability and performance of storage and supercomputer systems. Parallel filesystems (PFSs) are the most critical high-availability components of High Performance Computing (HPC) architectures, providing input/output (I/O) services to running computations, the environment that users and system services operate in, and storage for applications and data. Because of this central role, failure or performance degradation events in the PFS impact every user of an HPC resource. PFS events must be dealt with quickly and effectively by system administrators; however, there is typically insufficient information to establish precise causal relationships between PFS activity and events, impeding the implementation of timely and targeted remedies. To fill this information gap, an open-source Parallel FileSystem TRacing and Analysis SErvice (PFSTRASE) that traces and analyzes the requisite data to establish causal relationships between PFS activity and both realized and imminent events will be developed. This project will implement the service for the open-source Lustre filesystem, which is the most commonly used PFS at large-scale HPC sites. Loads for specific PFS directory and file operations will be measured and incorporated into the service to construct authentic server load contributions from every job, process, and user. The service?s infrastructure will continuously monitor the entire PFS and generate a real-time, seamless representation that connects contributions of jobs, processes, and users to storage server loads, network bandwidth, and storage capacities. The infrastructure will provide an easily navigable web interface that presents this data, both real-time and historical, in a visual format.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目将开发一个开源软件服务，并行文件系统跟踪和分析服务(PFSTRASE)，以提高国家-S最大的超级计算机-数据存储系统的可靠性和性能。随着模拟和计算更真实地代表现实，它们的规模随着它们消耗和生成的数据的大小而相应增长。为了处理这些数据的存储和移动，超级计算系统构建在大规模并行数据存储系统的主干上。由于它们的并行特性，这些存储系统能够以传统存储系统数百倍的速度移动数据，从而实现其他不切实际的计算。这些存储系统提供的性能功能伴随着复杂性，导致它们的运行通常远低于最佳运行状态，甚至在某些情况下会出现故障。这会浪费计算时间，最终导致科学进步的损失。对于当前和未来的计算系统来说，能够揭示这些问题并提高存储系统可靠性和性能的工具的发展状况是不够的。PFSTRASE将通过持续和自动监控存储系统运行状况和性能来填补这一空白，并通过易于使用的界面提供洞察，从而提高存储和超级计算机系统的可靠性和性能。并行文件系统(PFS)是高性能计算(HPC)体系结构中最关键的高可用性组件，为正在运行的计算、用户和系统服务运行的环境以及应用程序和数据的存储提供输入/输出(I/O)服务。由于这一核心角色，PFS中的故障或性能降级事件会影响HPC资源的每个用户。PFS事件必须由系统管理员快速有效地处理；然而，通常没有足够的信息来确定PFS活动与事件之间的确切因果关系，从而阻碍了及时和有针对性的补救措施的实施。为了填补这一信息空白，将开发一个开源并行文件系统跟踪和分析服务(PFSTRASE)，该服务跟踪和分析必要的数据，以建立PFS活动与已实现和即将发生的事件之间的因果关系。该项目将为开源的Lustre文件系统实现服务，该文件系统是大型HPC站点最常用的PFS。将测量特定PFS目录和文件操作的负载并将其合并到服务中，以构建来自每个作业、进程和用户的可信服务器负载贡献。服务-S基础架构将持续监视整个PFS，并生成实时无缝表示，将作业、进程和用户的贡献与存储服务器负载、网络带宽和存储容量联系起来。该基础设施将提供一个易于导航的网络界面，以可视化的形式呈现这些数据，包括实时和历史数据。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Democratizing Parallel Filesystem Monitoring

并行文件系统监控民主化

• DOI: 10.1109/cluster49012.2020.00065
• 发表时间: 2020
• 期刊: 2020 IEEE International Conference on Cluster Computing (CLUSTER
• 作者: Evans, Richard Todd