SDCI HPC: Improvement: Parallel I/O Software Infrastructure for Petascale Systems

技术优点：该项目旨在解决千万亿级并行机器的软件问题，它特别针对可扩展的I/O、存储和具有深度内存层次访问的系统。特别是，该项目建议改进、增强、开发和部署健壮的软件基础设施，以提供端到端的可伸缩I/O性能，这种性能利用对高级访问模式（意图）的理解，并通过运行时层使用该信息来实现不同级别的优化。我们提出了允许不同软件层相互交互和合作的机制，以实现端到端的性能目标。具体来说，这个项目的目标是开发、改进和部署(1)可扩展的软件，用于端到端I/O性能优化；(2)并行netCDF （PnetCDF）增强，提供统计功能和数据挖掘功能；(3)使用非阻塞I/O机制的PnetCDF软件优化；(4) MPI-IO缓存机制优化I/O软件栈性能；(5) I/O转发和专用缓存机制对有效利用即将到来的千兆级系统的结构很重要；(6)有效的I/O堆栈基准测试和测试套件；(7)优化辅助工具，通过程序分析识别并指导用户优化I/O；(8)利用作为NMI一部分开发的机制和工具进行测试；(9)帮助应用科学家将这些I/O堆栈优化整合到生产应用程序中的教程和工具。我们亦相信，本计划所开发的软件和技术，可直接应用于其他高级软件库和格式，例如层次数据格式（HDF）。更广泛的影响：我们将建立并利用我们团队的集体经验（包括广泛使用和强大的HPC软件系统的分发，如ROMIO， MPICH2， PVFS，PnetCDF和NU-Minebench）来分发在这个项目中为网络基础设施开发的软件，因此，直接影响许多领域应用程序的可扩展性。通过我们团队在多个基础设施中心（例如，teragrid）的积极参与，我们将在生产系统上部署软件。我们还将把这个项目的成果和经验纳入到我们的团队成员在并行计算、并行I/O和系统软件领域的各种教程中，这些教程将在世界上最重要的高性能计算会议上发表。通过这个项目和利用暑期实习，我们将为学生提供与应用科学家一起工作的机会，从而促进跨学科的合作。该项目还将支持研究生攻读高级学位。PI Choudhary拥有超过23个博士学位，其中许多人加入了学术界和国家实验室。在这个项目中，许多pi已经毕业了几位女性博士，我们将继续加强这一传统。除了将这个项目的课程纳入各种教程之外，我们还将像过去一样将它们纳入本科和研究生课程的课堂材料中。最后，我们与高性能计算领域的工业界有着强有力的合作，我们将利用这种合作向他们提供这个项目的成果和成果。

Technical Merit: This project proposes to address the software problem for petascale parallel machines, and it especially targets for scalable I/O, storage and systems with deep memory hierarchy accesses. In particular,this project proposes to improve, enhance, develop, and deploy robust software infrastructure to provide end-to-end scalable I/O performance that utilizes the understanding of high-level access patterns (?intent?), and uses that information through runtime layers to enable optimizations at different levels. We propose mechanisms that allow different software layers to interact and cooperate with each other to achieve end-to-end performance objectives. Specifically, the objectives of this project, are to develop, improve and deploy (1) scalable software for end-to-end I/O performance optimizations; (2) Parallel netCDF (PnetCDF) enhancements providing statistical functions and data mining functions; (3) PnetCDF software optimizations using non-blocking I/O mechanisms; (4) MPI-IO caching mechanisms to optimize I/O software stackperformance; (5) I/O forwarding and dedicated caching mechanisms important to effectively utilize the structures of upcoming petascale systems; (6) effective benchmarking and testing suites for the I/O stack; (7) an optimization assist tool that, through program analysis, can identify and guide a user to optimize I/O; (8) testing leveraging the mechanisms and tools developed as part of the NMI; and (9) tutorials and tools for helping application scientists incorporate these I/O stack optimizations into their production applications. We also believe that the software and techniques developed in this project will be directly applicable to and useful in other high-level software libraries and formats such as the Hierarchical Data Format (HDF).Broader Impact: We will build upon and leverage our team's collective experience (which includes distribution of widely used and robust software systems for HPC such as ROMIO, MPICH2, PVFS,PnetCDF and NU-Minebench) to distribute software developed in this project for cyberinfrastructure, andtherefore, directly impact the scalability of applications in many domains. Through our team's active participation in multiple infrastructure centers (e.g., teragrid), we will deploy the software on production systems. We will also incorporate the results and lessons from this project into the various tutorials that are presented by our team members in the area of parallel computing, parallel I/O and systems software in most leading conferences in HPC throughout the world. Through this project and utilizing summer internships, wewill provide an opportunity to students to work with application scientists, thereby fostering interdisciplinary collaboration. This project will also support graduate students work towards advanced degrees. PI Choudhary has graduated more than 23 PhDs, many of whom have joined academia and national labs. Multiple PIs in this project have graduated several female and underrepresented PhDs, and we will continue to enhance this tradition. In addition to incorporating the lessons from this project into various tutorials, we will also incorporate them into classroom material both for undergraduate and graduate level courses as we have done in the past. Finally, we have a strong collaboration with industry in the HPC area and we will leverage thatcollaboration to provide the outcomes and results of this project to them.