Hardware Acceleration of Co-Simulation for the Study of Extreme Weather Events

这是一项对日本京都大学灾害预防研究所(DPRI)进行为期两个月的研究访问的建议，目的是与大气和水球灾害部门的Tetsuya Takemi教授合作，研究极端天气事件联合模拟的硬件加速。特别是，我们的目标是加快天气研究和预报(WRF)模型和定制模拟器(如大型涡流模拟器)的联合模拟，并将组合模拟的运行时间减少一个数量级。运行时间的减少将使科学家能够以更高的精度进行极端天气事件的模拟。这次访问是我们在2012年访问的后续访问，那次访问建立了合作关系，并导致在HPCS会议上发表了一篇文章。**项目的重点*数值天气预测模型拟议中将使用的特定数值天气预报应用程序是：-天气研究和预测模型(WRF)。这是气候研究中的领先模型。它是一个开源(http://wrf-model.org/))的下一代中尺度数值天气预报系统，旨在服务于业务预报和大气研究需求。然而，由于其设计的复杂性，WRF模型目前不使用硬件加速(除了少量的实验模块)。-大型涡流模拟器是在DPRI专门开发的，用于研究飓风等恶劣天气事件对城市地区的影响。*联合模拟项目的重点是联合模拟，这是一种两个模拟器并行运行，一个模拟器的输出作为另一个模拟器的输入的方法。联合模拟是实现更有效的数值预报模拟的一个非常重要的机制：许多科学家开发了定制的模拟器，但是这些模拟器依赖于现有模拟器的输入，例如WRF。尤其是在研究恶劣天气事件时，科学家们想要改变控制方程：由于系统的复杂性，修改WRF核心通常不是一种选择。然而，当前的方法需要运行WRF并将运行结果写入文件，然后将生成的数据读取到定制模拟器中，这是非常无效的，因为需要生成大量数据并将它们存储在硬盘上。硬盘访问通常比内存访问慢1000倍。在模拟器的每个时间步长都必须从磁盘读取输入数据，导致操作非常慢。在联合仿真中，第一仿真器(WRF)产生的数据在每个时间步通过存储器传输到第二仿真器(例如LES)。*加快进程WRF模型的代码非常复杂。我们已经在中表明了加速WRF的空间，但要有一个完全加速的WRF版本需要几年的时间。因此，对于这次研究访问，我们计划使用当前形式的WRF，并使用MPI在计算集群上运行它，这是运行WRF的最有效方式。然而，在实践中创建支持GPU的小型定制模拟器是可行的，专家可以在几周内完成这项工作(例如，我们已经创建了支持GPU的LES版本)。通过在GPU或其他加速器上运行第二个仿真器，我们以WRF仿真的速度实现了完全协同仿真。在研究访问期间，我们希望创建一个系统，使WRF和GPU加速的模拟器之间的联合模拟成为可能。

This is a proposal for a two-month research visit to the Disaster Prevention Research Institute (DPRI) of the University of Kyoto in Japan, to work with Prof. Tetsuya Takemi of the Atmospheric and Hydrospheric Disasters Division on hardware acceleration of co-simulation of extreme weather events. In particular, our aim is to accelerate co-simulation of the Weather Research and Forecasting (WRF) model and custom simulators such as the Large Eddy Simulator, and to reduce the run times for combined simulations by an order of magnitude. This reduction in run time will allow scientists to perform simulations of extreme weather events at much higher precision.This visit is a follow-on visit from our previous visit in 2012, which established to collaboration and led to a publication at the HPCS conference.** Focus of the Project * Numerical Weather Prediction ModelsThe particular NWP applications to be used in the proposed work are:- The Weather Research and Forecasting Model (WRF). This is the leading model in climate research. It is an open-source (http://wrf-model.org/) next-generation mesoscale numerical weather prediction system designed to serve both operational forecasting and atmospheric research needs. However, because of the complexity of its design, the WRF model currently does not make use of hardware acceleration (except for a small number of experimental modules).- The Large Eddy Simulator is developed at the DPRI specifically to study the effects of severe weather events such as hurricanes on urban areas. * Co-simulationThe focus of the project is co-simulation, an approach where two simulators run in parallel and the outputs of one simulator serve as inputs for the other. Co-simulation is a very important mechanism to achieve more efficient NWP simulations: many scientists develop custom simulators that however rely on inputs from existing simulators such as WRF. This is in particular the case in the study of severe weather events where the scientists want to change the governing equations: modifying the WRF core is generally not an options because of the complexity of the system. However, the current approach, which involves running WRF and writing the results of the run to a file, then reading the generated data into the custom simulator, is extremely ineffective because of the need to generate huge amounts of data and store them on hard disks. Hard disk access is typically 1000x slower than memory access. Having to read input data from disk at every time step of the simulator results in very slow operation. In co-simulation, the data generated by the first simulator (WRF) is transferred via memory to the second simulator (e.g. LES) at every time step. * Accelerating the processThe code for the WRF model is very complex. We have shown in that there is scope for accelerating WRF, but it will take several years to have a fully accelerated version of WRF. Consequently, for this research visit, we plan to use WRF in its current form, and run it on a compute cluster using MPI, which is the most efficient way to run WRF. However, in practice creating a GPU-capable version of a small custom simulator is feasible and an expert can do this in a few weeks (we have for example already created a GPU-capable version of the LES). By running the second simulator on a GPU or other accelerator, we achieve full co-simulation at the speed of the WRF simulation. During the research visit, we want to create the system that will make co-simulation between WRF and a GPU-accelerated simulator possible.