BIGDATA: IA: Democratizing Massive Fluid Flow Simulations via Open Numerical Laboratories and Applications to Turbulent Flow and Geophysical Modeling

紊流的计算机模拟在工程应用（例如减少车辆的阻力和预测风力涡轮机的气动效率）和地球物理科学（例如描述污染物扩散或拉格朗日输运和海洋混合的命运）中发挥着越来越重要的作用。模拟包括离散和积分控制流体流动和在时间上向前传输的偏微分方程，在整个感兴趣的域内提供物理变量（如速度和压力）作为时间和空间函数的解。由于这样的模拟产生了大量的数据，研究人员普遍采用的方法是在模拟运行期间“动态”分析数据，而只存储一小部分时间步长以供后续分析。因此，在出现最初并不明显的新问题后，通常必须重复对同一过程进行大规模模拟。许多（甚至大多数）突破性概念无法提前预测，因为它们将部分由输出数据驱动，然后必须对其进行测试。因此，需要一种方法来存储来自这种模拟的整个时空数据。该项目开发了创新的工具，用于有效地创建开放的数值数据库，这些数据库包含湍流研究和地球物理传输建模中使用的计算流体动力学模拟的大量输出，并使这些数据库可供整个社区使用。开放数值实验室中包含的一些数据集将由外部研究人员提供。除了加强工程和地球物理流体力学和湍流研究外，大规模湍流模拟数据的民主化访问也将在下一代研究人员的教育和培训中发挥关键作用。主动学习通过新的教育模块，允许学生查询模拟数据集前所未有的细节将提供新的教育范式。更广泛地说，从这个项目中吸取的经验教训将推广到许多其他领域，在这些领域中，数值模拟产生了非常大的数据集，使用现行方法难以访问。通过这种方式，该项目将增强美国高性能科学计算基础设施的科学和更广泛的影响。该项目将开发创新工具，用于有效创建开放的数值数据库，该数据库包含用于湍流研究和地球物理传输建模的计算流体动力学模拟的大量输出。即将开发的摄取管道将使用户能够从包含大量直接数值模拟输出的文件系统中传输数据，建立数据库，并将其提供给社区，用于开放的探索性数据分析和创新的湍流和海洋混合研究。迄今为止，参与该项目的研究人员已经建立了一个开放数值实验室，专注于典型湍流的直接数值模拟（DNS），其中整个时空数据可供更广泛的研究团体使用。然而，现有的数据集数量很少，数据库是一个接一个创建的，使用的方法很难大规模复制。此外，新兴的百亿亿次模拟可能会产生前所未有规模的数据集（数十到数百pb）。解决这些挑战需要先进的计算机科学算法。该项目将(a)开发自动化的、可扩展的数据管理算法，以获取、索引和服务由广泛群体生成的非常大的数据集，(b)探索使用时空子采样结合在线插值和重新模拟的新算法，根据子采样步幅产生较大的压缩因子。(c)使用机器学习算法识别模拟中感兴趣的局部区域，并将这些4D域保存在数据库中，以便进行详细的后续分析。新的数据库将包括以下数据：(1)最大通道流DNS，(2)地球物理感兴趣的旋转和分层湍流，(3)发展壁面边界层的DNS，以及(4)具有复杂边界条件的详细海洋环流模型。作为创新领域科学应用的一部分，数据集将用于使用数据同化概念改进湍流模型，研究拉格朗日涡旋动力学，并探索北大西洋区域环流模式中的地球物理传输。

Computer simulations of turbulent fluid flows are playing an increasingly vital role in engineering applications (e.g. reducing drag forces on vehicles and predicting wind turbine aerodynamic efficiency) and in geophysical sciences (e.g. describing the fate of pollutant dispersion or Lagrangian transport and mixing in the ocean). Simulations consist of discretizing and integrating the partial differential equations governing fluid flow and transport forward in time, providing solutions for physical variables (fields such as velocity and pressure) as function of time and space in the entire domain of interest. Since such simulations generate enormous amounts of data, the prevailing approach has been for researchers to analyze the data "on the fly" during the simulation runs while only a small subset of time-steps are stored for subsequent analysis. As a result, often large simulations of the same process must be repeated after new questions arise that were not initially obvious. Many (or even most) breakthrough concepts cannot be anticipated in advance, as they will be motivated in part by output data and must then be tested against it. As a result, there is a need for methods to store entire space-time data from such simulations. This project develops innovative tools for the efficient creation of open numerical databases that contain massive outputs from computational fluid dynamics simulations used in turbulence research and geophysical transport modeling and makes these available to the entire community. Several of the datasets to be included into the Open Numerical Laboratory will be contributed by external researchers. In addition to enhancing engineering and geophysical fluid mechanics and turbulence research, democratized access to large-scale turbulent flow simulation data will also play a crucial role in education and training for the next generation of researchers. Active learning through new educational modules that allow students to query simulation datasets in unprecedented detail will provide new educational paradigms. More broadly, the lessons learned from this project will be generalizable to many other fields where numerical simulations generate very large datasets that are difficult to access using prevailing approaches. In this way, the project will enhance the scientific and broader impacts of the US high-performance scientific computing infrastructure. This project will develop innovative tools for the efficient creation of open numerical databases that contain massive outputs from computational fluid dynamics simulations used in turbulence research and geophysical transport modeling. An ingest pipeline to be developed will enable users to transfer data from file systems containing the output of their massive direct numerical simulations, build a database, and serve it to the community for open exploratory data analysis and innovative turbulence and oceanic mixing research. To date, the investigators involved in this project have built an Open Numerical Laboratory focusing on direct numerical simulations (DNS) of canonical turbulent flows, in which the entire space-time data are available to the wider research community. However, the existing datasets are few in number and databases have been created one by one, using methodologies difficult to replicate on a massive scale. Moreover, emerging Exascale simulations will potentially result in data sets of unprecedented scale (tens to hundreds of PetaBytes). Advanced computer science algorithms will be required to tackle these challenges. This project will (a) develop automated, and scalable data management algorithms to ingest, index and serve very large data sets generated by a wide range of groups, (b) explore novel algorithms using spatio-temporal subsampling combined with online interpolation with re-simulation, yielding large compression factors depending on the subsampling stride, and (c) use machine learning algorithms to identify localized regions of interest in the simulations and save these 4D domains in a database for detailed follow-up analytics. The new databases will include data from (1) the largest channel flow DNS, (2) rotating and stratified turbulence of geophysical interest, (3) a DNS of developing wall boundary layer and (4) detailed ocean circulation models with complex boundary conditions. As part of the innovative domain science applications, data sets will be used to improve turbulence models using data-assimilation concepts, study Lagrangian vortex dynamics, and explore geophysical transport in a regional general circulation model of the North Atlantic Ocean.