SDCI HPC: Improvement: Parallel I/O Software Infrastructure for Petascale Systems

技术优点：本项目旨在解决千万亿次并行机的软件问题，特别是针对可扩展的I/O、存储和具有深存储层次访问的系统。特别是，该项目建议改进，增强，开发和部署强大的软件基础设施，以提供端到端的可扩展的I/O性能，利用高层访问模式的理解（？意图？），并通过运行时层使用该信息来实现不同级别的优化。我们提出的机制，允许不同的软件层相互作用和相互合作，以实现端到端的性能目标。具体来说，该项目的目标是开发、改进和部署（1）用于端到端I/O性能优化的可扩展软件;（2）提供统计功能和数据挖掘功能的并行netCDF（PnetCDF）增强功能;（3）使用非阻塞I/O机制的PnetCDF软件优化;（4）MPI-IO缓存机制，以优化I/O软件堆栈性能;（5）I/O转发和专用高速缓存机制，其对于有效地利用即将到来的千万亿次系统的结构是重要的;（6）用于I/O栈的有效基准测试和测试套件;（7）优化辅助工具，其通过程序分析可以识别和引导用户优化I/O;（8）利用作为NMI的一部分开发的机制和工具进行测试;（9）帮助应用科学家将这些I/O堆栈优化纳入其生产应用程序的教程和工具。我们还相信，本项目中开发的软件和技术将直接适用于其他高级软件库和格式，如分层数据格式（HDF）。我们将利用我们团队的集体经验（其中包括广泛使用的和强大的HPC软件系统，如ROMIO，MPICH 2，PVFS，PnetCDF和NU-Minebench）分发该项目中为网络基础设施开发的软件，因此，直接影响许多领域应用程序的可扩展性。通过我们的团队积极参与多个基础设施中心（例如，teragrid），我们将在生产系统上部署该软件。我们还将把该项目的成果和经验教训融入到我们的团队成员在世界各地的HPC主要会议上介绍的并行计算、并行I/O和系统软件领域的各种教程中。通过这个项目和利用暑期实习，我们将提供一个机会，学生与应用科学家合作，从而促进跨学科的合作。该项目还将支持研究生攻读高级学位。PI Choudhary已经毕业了超过23个博士学位，其中许多人加入了学术界和国家实验室。在这个项目中的多个PI已经毕业了几个女性和代表性不足的博士学位，我们将继续加强这一传统。除了将这个项目的经验教训纳入各种教程，我们还将把它们纳入课堂材料都为本科和研究生水平的课程，因为我们已经在过去做的。最后，我们在HPC领域与业界有着密切的合作，我们将利用这种合作向他们提供该项目的成果和结果。

Technical Merit: This project proposes to address the software problem for petascale parallel machines, and it especially targets for scalable I/O, storage and systems with deep memory hierarchy accesses. In particular,this project proposes to improve, enhance, develop, and deploy robust software infrastructure to provide end-to-end scalable I/O performance that utilizes the understanding of high-level access patterns (?intent?), and uses that information through runtime layers to enable optimizations at different levels. We propose mechanisms that allow different software layers to interact and cooperate with each other to achieve end-to-end performance objectives. Specifically, the objectives of this project, are to develop, improve and deploy (1) scalable software for end-to-end I/O performance optimizations; (2) Parallel netCDF (PnetCDF) enhancements providing statistical functions and data mining functions; (3) PnetCDF software optimizations using non-blocking I/O mechanisms; (4) MPI-IO caching mechanisms to optimize I/O software stackperformance; (5) I/O forwarding and dedicated caching mechanisms important to effectively utilize the structures of upcoming petascale systems; (6) effective benchmarking and testing suites for the I/O stack; (7) an optimization assist tool that, through program analysis, can identify and guide a user to optimize I/O; (8) testing leveraging the mechanisms and tools developed as part of the NMI; and (9) tutorials and tools for helping application scientists incorporate these I/O stack optimizations into their production applications. We also believe that the software and techniques developed in this project will be directly applicable to and useful in other high-level software libraries and formats such as the Hierarchical Data Format (HDF).Broader Impact: We will build upon and leverage our team's collective experience (which includes distribution of widely used and robust software systems for HPC such as ROMIO, MPICH2, PVFS,PnetCDF and NU-Minebench) to distribute software developed in this project for cyberinfrastructure, andtherefore, directly impact the scalability of applications in many domains. Through our team's active participation in multiple infrastructure centers (e.g., teragrid), we will deploy the software on production systems. We will also incorporate the results and lessons from this project into the various tutorials that are presented by our team members in the area of parallel computing, parallel I/O and systems software in most leading conferences in HPC throughout the world. Through this project and utilizing summer internships, wewill provide an opportunity to students to work with application scientists, thereby fostering interdisciplinary collaboration. This project will also support graduate students work towards advanced degrees. PI Choudhary has graduated more than 23 PhDs, many of whom have joined academia and national labs. Multiple PIs in this project have graduated several female and underrepresented PhDs, and we will continue to enhance this tradition. In addition to incorporating the lessons from this project into various tutorials, we will also incorporate them into classroom material both for undergraduate and graduate level courses as we have done in the past. Finally, we have a strong collaboration with industry in the HPC area and we will leverage thatcollaboration to provide the outcomes and results of this project to them.