Using On-Demand File Systems in HPC Environments
Using On-Demand File Systems in HPC Environments
复制标题
在 HPC 环境中使用按需文件系统
DOI：
--
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
A. Streit
中科院分区：
文献类型：
--
作者：
Mehmet Soysal;M. Berghoff;T. Zirwes;Marc;Sebastian Oeste;A. Brinkmann;W. Nagel;A. Streit
在现代HPC系统中，并行(分布式)文件系统用于实现对存储基础架构的快速访问。然而，大规模HPC系统的I/O性能已经跟不上计算能力的增长。因此，I/O子系统也必须应对大量要求苛刻的元数据操作，这往往是整个HPC系统的瓶颈。在某些情况下，即使是一个表现不佳的应用程序也可能导致整个HPC系统变慢，从而扰乱使用相同I/O子系统的其他应用程序。未来，随着更大、更强大的HPC系统的出现，这类情况可能会变得更加频繁。在这项工作中，我们为具有非常高I/O需求的应用程序提供了一个简单的解决方案。我们建议的解决方案是为HPC作业按需创建私有并行文件系统，并使用节点本地存储设备，例如固态磁盘(SSD)。我们展示了此功能很容易添加到现有的HPC环境中，并且只需要对系统进行最低限度的配置。我们的结论是，对正在运行的应用程序的影响是可控的，生成高负载的应用程序的好处大于坏处。我们发现，在某些情况下，应用程序的运行速度可能会较慢，但在这些情况下，全局文件系统上的负载会得到降低。
In modern HPC systems, parallel (distributed) file systems are used to allow fast access from and to the storage infrastructure. However, I/O performance in large-scale HPC systems has failed to keep up with the increase in computational power. As a result, the I/O subsystem which also has to cope with a large number of demanding metadata operations is often the bottleneck of the entire HPC system. In some cases, even a single bad behaving application can be held responsible for slowing down the entire HPC system, disrupting other applications that use the same I/O subsystem. These kinds of situations are likely to become more frequent in the future with larger and more powerful HPC systems. In this work, we present a simple solution for applications with very high I/O demands. Our proposed solution is to create a private parallel file system on-demand for an HPC job and use the node-local storage devices, e.g. solid-state-disks (SSD). We show that this feature is easy to add to an existing HPC environment and requires only minimal configuration to the system. We conclude that the impact on running applications is manageable and the advantages to applications that generate a high load outweigh the disadvantages. We show that in some cases applications may run slower, but the reduction of load on the global file system is prevailing in these cases.