On the Strong Scaling of the Spectral Element Solver Nek5000 on Petascale Systems
On the Strong Scaling of the Spectral Element Solver Nek5000 on Petascale Systems
复制标题
论千万亿次系统上谱元求解器 Nek5000 的强扩展
DOI：
10.1145/2938615.2938617
复制
发表时间：
2016
期刊：
影响因子：
--
通讯作者：
P. Schlatter
中科院分区：
文献类型：
--
作者：
N. Offermans;O. Marin;Michel Schanen;Jing Gong;P. Fischer;P. Schlatter
目前的工作是针对执行一个强大的缩放研究的高阶谱元素流体动力学求解器Nek 5000。之前的研究，如[5]，从理论的角度表明了一个可扩展性强的度量标准，我们在三个具有不同性能特征和互连网络的并行机上进行了广泛的测试，即Mira（IBM Blue Gene/Q），Beskow（Cray XC 40）和Titan（Cray XK 7）。模拟所考虑的测试用例对应于四种不同摩擦雷诺数Reτ = 180、360、550和1000时直管中的湍流。考虑到并行通信的线性模型，我们量化了机器特性，以便更好地评估代码的缩放行为。随后，使用采样和分析工具来测量大范围计算核心上的计算和通信时间。我们还研究了两个粗网格求解器XXT和AMG对计算时间的影响。超线性缩放，由于减少缓存未命中的观察在每台计算机上。在Mira上，每个核心的自由度大约为5000- 10，000，在Beskow上为30，000 - 50，0000，两台机器的问题大小影响很小，范围在10，000和220，000之间，具体取决于Titan上的问题大小。这项工作旨在为Nek 5000用户提供参考，并作为社区迈向亿亿次超级计算机时解决潜在问题的基础。
The present work is targeted at performing a strong scaling study of the high-order spectral element fluid dynamics solver Nek5000. Prior studies such as [5] indicated a recommendable metric for strong scalability from a theoretical viewpoint, which we test here extensively on three parallel machines with different performance characteristics and interconnect networks, namely Mira (IBM Blue Gene/Q), Beskow (Cray XC40) and Titan (Cray XK7). The test cases considered for the simulations correspond to a turbulent flow in a straight pipe at four different friction Reynolds numbers Reτ = 180, 360, 550 and 1000. Considering the linear model for parallel communication we quantify the machine characteristics in order to better assess the scaling behaviors of the code. Subsequently sampling and profiling tools are used to measure the computation and communication times over a large range of compute cores. We also study the effect of the two coarse grid solvers XXT and AMG on the computational time. Super-linear scaling due to a reduction in cache misses is observed on each computer. The strong scaling limit is attained for roughly 5000-10,000 degrees of freedom per core on Mira, 30,000 - 50,0000 on Beskow, with only a small impact of the problem size for both machines, and ranges between 10,000 and 220,000 depending on the problem size on Titan. This work aims at being a reference for Nek5000 users and also serves as a basis for potential issues to address as the community heads towards exascale supercomputers.