Collaborative Research: An Application Driven I/O Optimization Approach for PetaScale Systems and Scientific Discoveries

本研究的重点是开发可扩展的并行文件访问方法的多尺度问题域分解，如自适应网格细化（AMR）为基础的算法。现有的并行I/O方法集中在一个相当均匀分布的请求模式下优化进程协作。然而，它们不适合AMR中的数据结构，因为底层数据分布是高度不规则和动态的。在现有的并行I/O方法中，如果进程协作没有仔细协调，则进程同步会对I/O并行性产生不利影响。本研究通过在并行netCDF库（PnetCDF）中开发可扩展的解决方案来解决此类同步问题，特别是解决AMR结构化数据及其I/O模式。PnetCDF是许多计算机仿真社区使用的流行I/O库。并行存储和访问AMR数据的可扩展解决方案被认为是一项具有挑战性的任务。本研究将设计一个基于进程组的并行I/O方法，以消除不相关的进程，从而避免可能的I/O串行化。此外，在pnetCDF中还将开发一种新的元数据表示法，以便以可移植的形式保存树形结构的AMR数据关系。

This research focuses on developing scalable parallel file access methods for multi-scale problem domain decompositions, such as the one presented in Adaptive Mesh Refinement (AMR) based algorithms. Existing parallel I/O methods concentrate on optimizing the process collaboration under a fairly evenly-distributed request pattern. However, they are not suitable for data structures in AMR, because the underlying data distribution is highly irregular and dynamic. Process synchronization in the existing parallel I/O methods can penalize the I/O parallelism if the process collaboration is not carefully coordinated. This research addresses such synchronization issue by developing scalable solutions in the Parallel netCDF library (PnetCDF), particularly to address AMR structured data and its I/O patterns. PnetCDF is a popular I/O library used by many computer simulation communities. A scalable solution for storing and accessing AMR data in parallel is considered a challenging task. This research will design a process-group based parallel I/O approach to eliminate unrelated processes and thus avoid possible I/O serialization. In addition, a new metadata representation will also be developed in pnetCDF for conserving tree-structured AMR data relationship in a portable form.