A Best Practice Analysis of HDF 5 5 and NetCDF- 4 4 Using Lustre
A Best Practice Analysis of HDF 5 5 and NetCDF- 4 4 Using Lustre
复制标题
使用 Lustre 对 HDF 5 5 和 NetCDF-4 4 的最佳实践分析
DOI：
10.1007/978-3-319-20119-1_20
复制
发表时间：
2015
期刊：
影响因子：
--
通讯作者：
T. Ludwig
中科院分区：
文献类型：
--
作者：
C. Bartz;K. Chasapis;M. Kuhn;P. Nerge;T. Ludwig
随着高性能计算（HPC）系统中核心数量的不断增加，应用程序产生更多的数据，这些数据最终必须并行存储和访问。HPC中应用程序的I/O以分层方式执行;科学应用程序使用标准化的高级库和数据格式（如HDF和NetCDF）来存储和操作位于并行文件系统中的数据。在本文中，我们提出了一个性能分析的并行接口的HDF和NetCDF-使用不同的测试配置，以提供最佳实践选择正确的I/O配置。我们的评估遵循分解方法，我们检查每一层的性能损失。测试的配置包括：（i）不同的访问模式，不相交和交错（ii）对齐和不对齐的访问（iii）集体和独立的I/O（iv）连续和分块的数据布局。主要观察结果是，在特定配置中使用交错数据访问可以实现接近最大性能的性能。此外，我们看到NetCDF-不提供将访问对齐到Lustre对象边界的能力。为了克服这个问题，我们开发了一个补丁，解决了这个问题，并大大提高了性能。
With the constantly increasing number of cores in high performance computing (HPC) systems, applications produce even more data that will eventually have to be stored and accessed in parallel. Applications’ I/O in HPC is performed in a layered manner; scientific applications use standardized high-level libraries and data formats like HDFand NetCDF-to store and manipulate data that is located inside a parallel file system. In this paper, we present a performance analysis of the parallel interfaces of HDFand NetCDF-using different test configurations in order to provide best practices for choosing the right I/O configuration. Our evaluation follows a breakdown approach where we examine the performance penalties of each layer. The tested configurations include:(i)different access patterns, disjoint and interleaved(ii)aligned and unaligned accesses(iii)collective and independent I/O(iv)contiguous and chunked data layout. The main observation is that using interleaved data access in a certain configuration achieves near the maximum performance. Also, we see that NetCDF-does not provide the ability to align the access to the Lustre object boundaries. To overcome this we have developed a patch that resolves this issue and improves the performance dramatically.