PIDX: Efficient Parallel I/O for Multi-resolution Multi-dimensional Scientific Datasets
PIDX: Efficient Parallel I/O for Multi-resolution Multi-dimensional Scientific Datasets
复制标题
PIDX：用于多分辨率多维科学数据集的高效并行 I/O
DOI：
10.1109/cluster.2011.19
复制
发表时间：
2011
期刊：
影响因子：
--
通讯作者：
R. Grout
中科院分区：
文献类型：
--
作者：
Sidharth Kumar;V. Vishwanath;P. Carns;B. Summa;G. Scorzelli;Valerio Pascucci;R. Ross;Jacqueline H. Chen;H. Kolla;R. Grout
IDX数据格式通过将数据存储在层次Z（HZ）顺序中，提供了有效，缓存遗忘和逐步访问大规模科学数据集的。以IDX格式存储的数据可以在交互式环境中可视化，从而允许使用最少的资源进行有意义的探索。这项技术可以实时，交互式可视化和大型数据集对从台式机和笔记本电脑到便携式设备（例如iPhone/iPad和Web）等各种系统的大型数据集。尽管用于编写IDX数据的现有访问API是串行的，但将IDX格式应用于大规模科学模拟的输出有明显的优势。因此，我们开发了PIDX-用于以IDX格式编写数据的并行API。使用PIDX，现在可以直接从大型科学模拟中生成IDX数据集，从而获得实时监视和可视化生成数据的额外优势。在本文中，我们提供了IDX文件格式的概述以及如何使用PIDX生成的概述。然后，我们提出一个数据模型描述和一种新颖的聚合策略，以增强PIDX库的可扩展性。 S3D燃烧应用被用作证明PIDX对现实世界科学模拟的功效的一个例子。 S3D用于对需要高保真度模拟的湍流燃烧的基本研究。 PIDX在8,192个过程中最多可实现18 GIB/S I/O的吞吐量，用于S3D以IDX格式写出数据。这允许对S3D数据进行互动分析和可视化，从而实现S3D模拟的原位分析。
The IDX data format provides efficient, cache oblivious, and progressive access to large-scale scientific datasets by storing the data in a hierarchical Z (HZ) order. Data stored in IDX format can be visualized in an interactive environment allowing for meaningful explorations with minimal resources. This technology enables real-time, interactive visualization and analysis of large datasets on a variety of systems ranging from desktops and laptop computers to portable devices such as iPhones/iPads and over the web. While the existing ViSUS API for writing IDX data is serial, there are obvious advantages of applying the IDX format to the output of large scale scientific simulations. We have therefore developed PIDX - a parallel API for writing data in an IDX format. With PIDX it is now possible to generate IDX datasets directly from large scale scientific simulations with the added advantage of real-time monitoring and visualization of the generated data. In this paper, we provide an overview of the IDX file format and how it is generated using PIDX. We then present a data model description and a novel aggregation strategy to enhance the scalability of the PIDX library. The S3D combustion application is used as an example to demonstrate the efficacy of PIDX for a real-world scientific simulation. S3D is used for fundamental studies of turbulent combustion requiring exceptionally high fidelity simulations. PIDX achieves up to 18 GiB/s I/O throughput at 8,192 processes for S3D to write data out in the IDX format. This allows for interactive analysis and visualization of S3D data, thus, enabling in situ analysis of S3D simulation.