AtmosFOAM parallel scaling on HECToR and New Test Cases which expose Grid Scale Oscillations

过去几十年里一些最好的天气和气候预报模型似乎不适用于未来几十年的计算机体系结构。计算机能力的进一步提高需要伴随着电力消耗的减少，这与硬件成本一起意味着，未来几代高性能计算机将由成千上万个更便宜、更弱的处理核心组成，就像电脑游戏中使用的那样。预测模型需要不同的设计，这样计算才能有效地分布在许多功能较弱的核心上。在设计将在下一代计算机上运行良好的新预测模型时，独立比较模型的不同方面是至关重要的，并且在整个模型组装之前能够严格测试简化模型版本是至关重要的。这两个方面将在本项目中分别涉及：(1)提出进一步开发一个简化的全球大气模型：AtmosFOAM，在英国高性能计算机HECToR的10,000个处理内核上运行。AtmosFOAM是使用OpenFOAM软件库编写的，该软件库已经在HECToR上运行。使用OpenFOAM编写的其他模型已经被证明可以很好地扩展到HECToR上的1,000或更多核。它的目标是达到15公里的水平空间分辨率和50个垂直水平，这意味着大约有1.5亿个计算点，每个核心上有1万到1.5万个计算点。这个问题的大小应该足够大，可以扩展到10,000个核心。并行缩放的两个瓶颈（由英国气象局确定）是半隐式时间步进，它允许更长的时间步进，但需要在所有处理核心之间进行同步的全球通信，以及在恒定经度线交汇的南北两极附近平滑解决方案。半隐式时间步进很重要，因为它意味着时间步长不必短到足以分辨声波。然而，如果在水平方向上使用显式时间步进，则需要更少的处理器通信，但需要更小的时间步进。在某种程度上，回到20世纪80年代的建模技术并使用在水平上明确的时间步进将是有益的。本项目将通过比较两种时间步进方案在其他方面相同的模型中的精度和并行性能，帮助找出数值算法何时必须进行这种改变。解决等经度线向两极收敛问题的一种方法是用球面的六边形-二十面体网格代替经纬度网格。这种镶嵌更像一个足球，但它把球体分成了更多的六边形。为了对两种网格进行清晰的比较，将使用其他方面相同的模型对两种网格类型的并行性能和精度进行比较。所有代码开发都将在Gnu公共许可证（GPL）下免费提供和开放源代码。(2)这项研究的一个独立部分将是为简化的大气模型创建新的测试用例，这些模型模拟现实世界中发生的非常小尺度的强活动，而这些活动在全球模式中无法解决。例如，雷暴和非常陡峭的山脉不能完全解决，但可以显著影响更大的尺度。在这些新的测试用例中，将在单点模拟这些影响的方程中添加项。这些强迫将在不适当的模型中产生网格尺度振荡或其他不现实的行为，否则这些模型可能会在其他现有的简单测试用例中表现良好。这可以防止业务中心采用不适当的模式。

Some of the best weather and climate forecasting models of the past few decades appear not to work well on the computer architectures of the coming decades. Further increases in computer power need to be accompanied by reduced electricity consumption and this, along with hardware cost, means that future generations of high performance computers will consist of a tens or hundreds of thousands of cheaper, less powerful processing cores, such as are used by computer games. Forecasting models need to be designed differently so that the computation can be distributed efficiently over so many less powerful cores. In designing new forecasting models that will perform well on next generation computers it is essential to compare different aspects of the model independently and it is essential to be able to test simplified model versions rigorously before the entire model is assembled. These two aspects will be covered separately in this project: (1) It is proposed to further develop a simplified model of the global atmosphere: AtmosFOAM, to run on 10,000 processing cores of the UK high performance computer, HECToR. AtmosFOAM is written using the OpenFOAM software library which has already been run on HECToR. Other models written using OpenFOAM have been shown to scale well to 1,000 or more cores on HECToR. It is aimed to reach a horizontal spatial resolution of 15km with 50 vertical levels meaning around 150 million computational points with 10 to 15 thousand points on each core. This problem size should be large enough to scale to 10,000 cores. Two of the bottlenecks to parallel scaling (identified by the Met Office) are semi-implicit time stepping which allows longer time steps but requires synchronised, global communication across all processing cores, and the smoothing of the solutions near the north and south poles where the lines of constant longitude converge. Semi-implicit time stepping is important because it means that the time step does not have to be short enough to resolve sound waves. However if explicit time stepping is used instead in the horizontal, less processor communication is needed but smaller time steps are needed. At some point it will become beneficial to go back to the modelling techniques of the 1980s and use time stepping which is explicit in the horizontal. This project will help to find out when this change in numerical algorithm must be made by comparing the accuracy and parallel performance of the two time stepping schemes in a model which is otherwise identical. A solution to the problem of the convergence of the lines of constant longitude towards the poles is to use a hexagonal-icosahedral grid of the sphere instead of a latitude-longitude grid. This tessellation is more like a football but it divides the sphere into many more hexagons. Parallel performance and accuracy will be compared on the two grid types using a model which is otherwise identical in order to give a clean comparison of the two grids. All code developments will be freely available and open source under the Gnu public licence (GPL). (2) An independent part of the research will be to create new test cases for simplified models of the atmosphere which mimic the strong activity at very small scales that occur in the real world which cannot be resolved in a global model. For example thunderstorms and very steep mountains cannot be fully resolved but can dramatically influence the larger scales. In these new test cases, terms will be added to the equations which mimic these influences at just a single point. These forcings will generate grid scale oscillations or other unrealistic behaviour in inadequate models which otherwise might appear to work well for other existing simple test cases. This could prevent inadequate models from being adopted by operational centres.