OAC Core: The Best of Both Worlds: Deep Neural Operators as Preconditioners for Physics-Based Forward and Inverse Problems

在科学、工程、医学和国防的各个领域，许多物理系统都用偏微分方程(PDE)高精度地建模，并在先进的计算系统上求解。通常最终目标是反复求解偏微分方程组，以探索参数的不确定性。出现这种情况的环境是逆问题(从数据推断模型的不确定参数)、最优实验设计(确定最佳数据采集以最大限度地了解模型)、最优设计(找到系统的最优配置以使性能最大化)和最优控制(确定系统的最优运行以实现所需的行为)。这些问题往往以高维不确定参数空间为特征，因为这些参数通常代表初始条件、边界条件、材料性质或源项，并且在空间和/或时间上变化。因此，为了充分表示参数中的不确定性，偏微分方程组往往需要求解数千次甚至数百万次。当被建模的系统涉及发生在多个空间和时间尺度上的耦合的多个物理或行为时，即使在最新的超级计算机上，重复求解PDE模型也变得令人望而却步。近年来发展起来的深度神经网络通过学习输入参数和输出参数(如温度、速度、压力、应力、电场、磁场、化学物质)之间的关系，有望克服偏微分方程模型重复求解的困难。一旦对PDE解决方案数据进行训练，网络就可以评估任何给定输入的输出，单位为毫秒，而求解PDE模型本身则需要数小时或数天。然而，尽管这些所谓的神经网络替代品的开发取得了很大进展，但它们通常只提供1-2位数的精度，这不足以取代PDE求解器。相反，这个项目正在开发神经网络替代品和PDE模型的混合体，它们结合了每种模型的最佳特性：PDE的准确性和神经网络的速度。其影响是，技术、健康、环境和社会中的许多问题现在将变得容易处理，这些问题不适合基于复杂模型的推理和决策。在这个项目中开发的算法将以开源软件的形式发布，以便广大研究人员和实践者可以将它们应用于一系列科学和工程问题。近年来，高保真偏微分方程解的神经网络近似，即神经算子，由于其易于实现、对不同设置的适应性以及似乎能够缓解维度诅咒而变得流行起来。最近的重要研究试图为各种类型的地图建立这些替代物的“通用近似”性质。虽然理论上表明，神经操作者原则上可以达到任意精度，但在实践中实现这一点仍然是一个巨大的挑战。其原因包括产生足够的训练数据的巨大成本，以及统计抽样误差、近似误差和训练问题的非凸性之间的混淆关系。与高保真的PDE解算器相比，神经操作员通常只能达到1-2位数的精度，几乎没有希望进一步降低这种精度。另一方面，高保真的偏微分方程组模型(特别是守恒定律和平衡律)通常具有很高的可信度，而且由于偏微分方程组的解对输入的微小扰动很敏感，因此需要高精度。神经运算符的中等精度通常不足以满足关键系统的推理、控制和决策的要求。这个项目正在开发神经算子和高保真PDE模型的混合，以实现每个模型的最佳特征，通过使用神经算子作为预条件来保持精度和速度。该项目的目标是用于偏微分方程组的线性和非线性参数神经预处理器，以及用于加速求解贝叶斯反问题的大都会朗之万法的神经预条件器。使用神经运算符作为预条件的另一个优势是它们可以很好地映射到GPU架构上。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Many physical systems across all areas of science, engineering, medicine, and defense are modeled with high accuracy by partial differential equations (PDEs) and solved on advanced computing systems. Often the ultimate goal is to repeatedly solve the PDEs to explore parameter uncertainties. Settings in which this arises are inverse problems (inferring uncertain parameters of a model from data), optimal experimental design (determining the optimal data acquisition to learn the most about the model), optimal design (finding the optimal configuration of a system to maximize performance), and optimal control (determining the optimal operation of a system to achieve a desired behavior). These problems are often characterized by high dimensional uncertain parameter spaces, since the parameters typically represent initial conditions, boundary conditions, material properties, or source terms and vary in space and/or time. As a result, the PDEs often have to be solved thousands or even millions of times to adequately represent uncertainties in the parameters. When the systems that are modeled involve coupled multiple physics or behavior occurring on multiple space and time scales, repeated solution of the PDE models becomes prohibitive, even on the latest supercomputers. The development of deep neural networks in recent years shows promise in overcoming the intractability of repeated solution of the PDE models, by learning the relationships between the input parameters and the outputs of interest (e.g., temperature, velocity, pressure, stress, electric field, magnetic field, chemical species). Once trained on PDE solution data, the networks can evaluate the outputs for any given inputs in milliseconds, compared to hours or days to solve the PDE models themselves. However, despite much progress in the development of these so-called neural network surrogates, they typically deliver just 1-2 digits of accuracy, which is not sufficient to replace the PDE solver. Instead, this project is developing hybrids of neural network surrogates and PDE models that combine the best properties of each: the accuracy of the PDEs with the speed of the neural networks. The impact is that many problems in technology, health, the environment, and society that were not amenable to complex model-based inference and decision making will now become tractable. The algorithms developed in this project are being released as open-source software so that a broad community of researchers and practitioners can apply them to a spectrum of scientific and engineering problems. In addition, the surrogate methods developed in this project are being incorporated into a popular graduate course on inverse problems taught at University of Texas, Austin.Neural network approximations of high fidelity PDE solutions, i.e., neural operators, have gained popularity in recent years due to their ease of implementation, adaptability to varied settings, and seeming ability to mitigate the curse of dimensionality. Significant recent research has attempted to establish "universal approximation" properties of these surrogates for various classes of maps. While theory suggests that neural operators can in principle achieve arbitrary accuracy, realizing this in practice remains a significant challenge. The reasons for this include the enormous costs of generating sufficient training data, and confounding relations between statistical sampling errors, approximation errors, and nonconvexity of the training problem. Often neural operators can achieve just 1-2 digits of accuracy relative to high fidelity PDE solvers, with little hope of further reducing this accuracy. On the other hand, high fidelity PDE models (particularly conservation and balance laws) are often known with very high confidence and high precision is necessary due to sensitivity of PDE solutions to small perturbations in the inputs. The modest accuracies of neural operators are often insufficient for the demands of inference, control, and decision making for critical systems. This project is developing hybrids of neural operators and high fidelity PDE models to realize the best features of each, by retaining accuracy via the PDE residual and speed via use of the neural operator as a preconditioner. The project targets linear and nonlinear parametric neural preconditioners for PDEs, and neural preconditioners for Metropolized Langevin methods to accelerate the solution of Bayesian inverse problems. A further advantage of using neural operators as preconditioners is that they map well onto GPU architectures.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.