SI2-SSE: GEM3D: Open-Source Cartesian Adaptive Complex Terrain Atmospheric Flow Solver for GPU Clusters

美国政府通过几个机构投资于领先的超级计算设施，以推进许多前沿的科学发现。这个项目的动机是国家对超级计算研究的承诺，以及从工作站到超级计算机的多核计算硬件的日益可用性。今天，科学家和工程师可以使用超大规模的计算资源。然而，许多遗留代码没有利用计算硬件的最新创新，并且缺乏能够有效利用多核计算范式的开源模拟科学软件。计算流体动力学（CFD）求解器推动了许多领域的发展，如航空航天工程和大气科学。目前许多开源的CFD模型和数值天气预报模型并没有充分利用图形处理单元（gpu）优越的计算性能。通过创建一个可以在多GPU工作站和大型GPU集群上执行的开源社区模型，项目团队希望扩大高性能计算在流体动力学应用中的应用。直接的目标应用是复杂地形上的风建模，以支持风资源评估、电力预测、大气研究和空气污染方面的研究和发展。通过这个项目，pi将继续向新学生传递和扩展GPU计算、计算数学和软件工程方面的知识库。在能够使用超级计算机进行科学研究的人才严重短缺的情况下，超越传统学科的技能组合受到国家实验室的高度重视。参与该项目的学生和博士后研究人员将为这一关键劳动力做出贡献。这个项目汇集了工程师、应用数学家和计算机科学家。整个软件组件套件将为GPU集群设计，采用MPI-CUDA实现，使用三维分解将计算与通信重叠，以增强可扩展性。实施将平衡性能和进一步的发展和更广泛的学术研究人员社区的所有权。团队将遵循并发应用程序的现代软件工程实践。将开发一种可在GPU集群上扩展的自适应网格细化策略。一种新的基于径向基函数的投影方法将对自适应精细网格层施加无发散约束。软件元素将使用并发程序的单元测试和验证技术，以及从基准数值问题中获得的数据进行测试。流动求解器将包括任意复杂地形的浸入边界法模块和动态大涡模拟技术模块。软件实现和语法将是直观的，以允许来自更大社区的贡献。项目团队希望该软件能够帮助减少高分辨率模拟的建模错误，并有助于对复杂地形上的湍流风有一个基本的了解。该项目的pi将继续他们在并行科学计算、计算数学和软件工程方面的教学工作。结果将通过会议演示和开源项目的wiki站点进行传播。软件元素将在开源GNU通用公共许可证下发布。

The U.S. Government invests in leadership supercomputing facilities through several agencies to advance scientific discovery in many fronts. This project is motivated by this national commitment to supercomputing research and the increasing availability of many-core computing hardware from workstations to supercomputers. Today scientists and engineers have access to extreme-scale computing resources. However, many legacy codes do not take advantage of recent innovations in computing hardware, and there is a lack of open-source simulation science software that can effectively leverage the many-core computing paradigm. Computational fluid dynamics (CFD) solvers have advanced many fields such as aerospace engineering and atmospheric sciences. Many current open-source CFD models and numerical weather prediction models do not take full advantage of the superior compute performance of graphics processing units (GPUs). By creating an open-source community model that can execute on multi-GPU workstations and large GPU clusters, the project team expects to broaden the use of high-performance computing in fluid dynamics applications. The immediate target application is wind modeling over complex terrain, to support research and development in wind resource assessment, power forecasting, atmospheric research, and air pollution. Through this project, the PIs will continue to transfer and expand the knowledge bases in GPU computing, computational mathematics, and software engineering to new students. Skill sets that transcend traditional disciplines are highly prized by national laboratories as there is a critical shortage of workforce who can conduct scientific research using supercomputers. Students and postdoctoral researchers who are involved in this project will contribute toward this critical workforce. This project brings together engineers, applied mathematicians, and computer scientists. The entire suite of software elements will be designed for GPU clusters with an MPI-CUDA implementation that overlaps computation with communications using a three-dimensional decomposition for enhanced scalability. The implementation will balance performance and further development and ownership by a broader community of academic researchers. The team will follow modern software engineering practices for concurrent applications. An adaptive mesh refinement strategy that can scale on GPU clusters will be developed. A novel projection method based on radial basis functions will impose the divergence-free constraint on a hierarchy of adaptively refined grids. Software elements will be tested using unit testing and verification techniques for concurrent programs, and against data available from benchmark numerical problems. The flow solver will include modules for the immersed boundary approach for arbitrarily complex terrain and the dynamic large-eddy simulation technique. The software implementation and syntax will be intuitive to allow contributions from a larger community. The project team expects the proposed software to help reduce modeling errors with very high resolution simulations and contribute toward a fundamental understanding of turbulent winds over complex terrain. The PIs of this project will continue their teaching efforts in Parallel Scientific Computing, Computational Mathematics, and Software Engineering. The results will be disseminated through conference presentations and via a wiki site for the open-source project. Software elements will be released under an open-source GNU General Public License.