SI2-SSE: GEM3D: Open-Source Cartesian Adaptive Complex Terrain Atmospheric Flow Solver for GPU Clusters

美国政府通过几个机构投资于Leadance超级计算设施，以推动多个战线的科学发现。该项目的动机是国家对超级计算研究的承诺，以及从工作站到超级计算机的多核计算硬件的可用性不断增加。今天，科学家和工程师可以使用极端规模的计算资源。然而，许多遗留代码没有利用最近在计算硬件方面的创新，并且缺乏能够有效利用多核计算范例的开源模拟科学软件。计算流体力学(CFD)解算器已经在航空航天工程和大气科学等许多领域取得了进展。当前许多开源的CFD模型和数值天气预报模型没有充分利用图形处理单元(GPU)的卓越计算性能。通过创建可以在多个GPU工作站和大型GPU集群上执行的开源社区模型，项目团队希望扩大高性能计算在流体动力学应用程序中的使用。直接的目标应用是复杂地形上的风模型，以支持在风资源评估、电力预测、大气研究和空气污染方面的研究和开发。通过这项计划，专业人员将继续向新生传递和扩展GPU计算、计算数学和软件工程方面的知识基础。超越传统学科的技能组合受到国家实验室的高度重视，因为能够使用超级计算机进行科学研究的劳动力严重短缺。参与该项目的学生和博士后研究人员将为这一关键的劳动力做出贡献。这个项目汇集了工程师、应用数学家和计算机科学家。整套软件元素将设计用于具有MPI-CUDA实施的GPU集群，该实施利用三维分解将计算与通信重叠，以增强可伸缩性。实施将平衡绩效和更广泛的学术研究人员社区的进一步发展和所有权。该团队将遵循并发应用程序的现代软件工程实践。将开发一种可在GPU集群上扩展的自适应网格细化策略。一种新的基于径向基函数的投影方法将无散度约束施加到自适应细化的网格层次上。软件元素将使用并发程序的单元测试和验证技术进行测试，并根据基准数值问题提供的数据进行测试。流动解算器将包括用于任意复杂地形的浸没边界方法和动态大涡模拟技术的模块。软件实现和语法将是直观的，以允许来自更大社区的贡献。该项目团队预计，拟议的软件将通过非常高分辨率的模拟来帮助减少建模错误，并有助于从根本上理解复杂地形上的湍流风。这个项目的专业人员将继续他们在并行科学计算、计算数学和软件工程方面的教学工作。结果将通过会议发言和通过开放源码项目的维基网站传播。软件元素将在开源GNU通用公共许可证下发布。

The U.S. Government invests in leadership supercomputing facilities through several agencies to advance scientific discovery in many fronts. This project is motivated by this national commitment to supercomputing research and the increasing availability of many-core computing hardware from workstations to supercomputers. Today scientists and engineers have access to extreme-scale computing resources. However, many legacy codes do not take advantage of recent innovations in computing hardware, and there is a lack of open-source simulation science software that can effectively leverage the many-core computing paradigm. Computational fluid dynamics (CFD) solvers have advanced many fields such as aerospace engineering and atmospheric sciences. Many current open-source CFD models and numerical weather prediction models do not take full advantage of the superior compute performance of graphics processing units (GPUs). By creating an open-source community model that can execute on multi-GPU workstations and large GPU clusters, the project team expects to broaden the use of high-performance computing in fluid dynamics applications. The immediate target application is wind modeling over complex terrain, to support research and development in wind resource assessment, power forecasting, atmospheric research, and air pollution. Through this project, the PIs will continue to transfer and expand the knowledge bases in GPU computing, computational mathematics, and software engineering to new students. Skill sets that transcend traditional disciplines are highly prized by national laboratories as there is a critical shortage of workforce who can conduct scientific research using supercomputers. Students and postdoctoral researchers who are involved in this project will contribute toward this critical workforce. This project brings together engineers, applied mathematicians, and computer scientists. The entire suite of software elements will be designed for GPU clusters with an MPI-CUDA implementation that overlaps computation with communications using a three-dimensional decomposition for enhanced scalability. The implementation will balance performance and further development and ownership by a broader community of academic researchers. The team will follow modern software engineering practices for concurrent applications. An adaptive mesh refinement strategy that can scale on GPU clusters will be developed. A novel projection method based on radial basis functions will impose the divergence-free constraint on a hierarchy of adaptively refined grids. Software elements will be tested using unit testing and verification techniques for concurrent programs, and against data available from benchmark numerical problems. The flow solver will include modules for the immersed boundary approach for arbitrarily complex terrain and the dynamic large-eddy simulation technique. The software implementation and syntax will be intuitive to allow contributions from a larger community. The project team expects the proposed software to help reduce modeling errors with very high resolution simulations and contribute toward a fundamental understanding of turbulent winds over complex terrain. The PIs of this project will continue their teaching efforts in Parallel Scientific Computing, Computational Mathematics, and Software Engineering. The results will be disseminated through conference presentations and via a wiki site for the open-source project. Software elements will be released under an open-source GNU General Public License.