Hardware Acceleration of Co-Simulation for the Study of Extreme Weather Events

这是一项对日本京都大学防灾研究所进行为期两个月的研究访问的提议，目的是与大气和水圈灾害司的Tetsuya Takemi教授就硬件加速共同模拟极端天气事件开展合作。特别是，我们的目标是加速天气研究和预报（WRF）模型和自定义模拟器（如大涡模拟器）的联合模拟，并将组合模拟的运行时间减少一个数量级。运行时间的减少将使科学家能够以更高的精度模拟极端天气事件。这次访问是我们2012年访问的后续访问，该访问建立了合作并在HPCS会议上发表了论文。项目重点 * 数值天气预报模式拟议工作中将使用的具体数值天气预报应用程序是：-天气研究和预报模式（WRF）。这是气候研究中的领先模型。这是一个开放源码（http：//wrf-model.org/）的下一代中尺度数值天气预报系统，旨在满足业务预报和大气研究的需要。然而，由于其设计的复杂性，WRF模型目前没有使用硬件加速（除了少数实验模块）。大涡模拟器是在DPRI开发的，专门用于研究飓风等恶劣天气事件对城市地区的影响。* 联合仿真该项目的重点是联合仿真，即两个仿真器并行运行，一个仿真器的输出作为另一个仿真器的输入。协同模拟是实现更高效NWP模拟的一个非常重要的机制：许多科学家开发了自定义模拟器，但依赖于现有模拟器（如WRF）的输入。在恶劣天气事件的研究中，科学家们希望改变控制方程：由于系统的复杂性，修改WRF核心通常不是一种选择。然而，当前的方法涉及运行WRF并将运行结果写入文件，然后将生成的数据阅读到自定义模拟器中，由于需要生成大量数据并将其存储在硬盘上，因此效率极低。硬盘访问通常比内存访问慢1000倍。必须在模拟器的每个时间步长从磁盘读取输入数据导致非常慢的操作。在协同仿真中，由第一仿真器（WRF）生成的数据在每个时间步经由存储器传送到第二仿真器（例如LES）。* WRF模型的代码非常复杂。我们在中已经展示了WRF加速的空间，但是要有一个完全加速的WRF版本还需要几年时间。因此，对于这次研究访问，我们计划以当前的形式使用WRF，并使用MPI在计算集群上运行它，这是运行WRF的最有效方法。然而，在实践中，创建一个支持GPU的小型自定义模拟器版本是可行的，专家可以在几周内完成（例如，我们已经创建了一个支持GPU的LES版本）。通过在GPU或其他加速器上运行第二个模拟器，我们以WRF模拟的速度实现了完全的协同模拟。在研究访问期间，我们希望创建一个系统，使WRF和GPU加速模拟器之间的联合仿真成为可能。

This is a proposal for a two-month research visit to the Disaster Prevention Research Institute (DPRI) of the University of Kyoto in Japan, to work with Prof. Tetsuya Takemi of the Atmospheric and Hydrospheric Disasters Division on hardware acceleration of co-simulation of extreme weather events. In particular, our aim is to accelerate co-simulation of the Weather Research and Forecasting (WRF) model and custom simulators such as the Large Eddy Simulator, and to reduce the run times for combined simulations by an order of magnitude. This reduction in run time will allow scientists to perform simulations of extreme weather events at much higher precision.This visit is a follow-on visit from our previous visit in 2012, which established to collaboration and led to a publication at the HPCS conference.** Focus of the Project * Numerical Weather Prediction ModelsThe particular NWP applications to be used in the proposed work are:- The Weather Research and Forecasting Model (WRF). This is the leading model in climate research. It is an open-source (http://wrf-model.org/) next-generation mesoscale numerical weather prediction system designed to serve both operational forecasting and atmospheric research needs. However, because of the complexity of its design, the WRF model currently does not make use of hardware acceleration (except for a small number of experimental modules).- The Large Eddy Simulator is developed at the DPRI specifically to study the effects of severe weather events such as hurricanes on urban areas. * Co-simulationThe focus of the project is co-simulation, an approach where two simulators run in parallel and the outputs of one simulator serve as inputs for the other. Co-simulation is a very important mechanism to achieve more efficient NWP simulations: many scientists develop custom simulators that however rely on inputs from existing simulators such as WRF. This is in particular the case in the study of severe weather events where the scientists want to change the governing equations: modifying the WRF core is generally not an options because of the complexity of the system. However, the current approach, which involves running WRF and writing the results of the run to a file, then reading the generated data into the custom simulator, is extremely ineffective because of the need to generate huge amounts of data and store them on hard disks. Hard disk access is typically 1000x slower than memory access. Having to read input data from disk at every time step of the simulator results in very slow operation. In co-simulation, the data generated by the first simulator (WRF) is transferred via memory to the second simulator (e.g. LES) at every time step. * Accelerating the processThe code for the WRF model is very complex. We have shown in that there is scope for accelerating WRF, but it will take several years to have a fully accelerated version of WRF. Consequently, for this research visit, we plan to use WRF in its current form, and run it on a compute cluster using MPI, which is the most efficient way to run WRF. However, in practice creating a GPU-capable version of a small custom simulator is feasible and an expert can do this in a few weeks (we have for example already created a GPU-capable version of the LES). By running the second simulator on a GPU or other accelerator, we achieve full co-simulation at the speed of the WRF simulation. During the research visit, we want to create the system that will make co-simulation between WRF and a GPU-accelerated simulator possible.