OAC Core: The Best of Both Worlds: Deep Neural Operators as Preconditioners for Physics-Based Forward and Inverse Problems

科学、工程、医学和国防领域的许多物理系统都是通过偏微分方程（PDE）高精度建模的，并在先进的计算系统上求解。通常，最终目标是反复求解偏微分方程，以探索参数的不确定性。其中出现的设置是逆问题（从数据推断模型的不确定参数），最优实验设计（确定最佳数据采集以了解最多的模型），最优设计（找到系统的最佳配置以最大化性能）和最优控制（确定系统的最佳操作以实现期望的行为）。这些问题的特点往往是高维的不确定参数空间，因为参数通常表示初始条件，边界条件，材料属性，或源项，并在空间和/或时间变化。因此，偏微分方程往往需要求解数千甚至数百万次才能充分表示参数中的不确定性。当被建模的系统涉及在多个空间和时间尺度上发生的耦合多个物理或行为时，PDE模型的重复求解变得令人望而却步，即使在最新的超级计算机上也是如此。近年来深度神经网络的发展显示出通过学习输入参数和感兴趣的输出之间的关系（例如，温度、速度、压力、应力、电场、磁场、化学物质）。一旦在PDE解决方案数据上进行了训练，网络就可以在毫秒内评估任何给定输入的输出，而不是在几小时或几天内解决PDE模型本身。然而，尽管这些所谓的神经网络代理的开发取得了很大进展，但它们通常只能提供1-2位数的精度，这不足以取代PDE求解器。相反，该项目正在开发神经网络代理和PDE模型的混合体，这些混合体联合收割机结合了两者的最佳特性：PDE的准确性和神经网络的速度。其影响是，技术、健康、环境和社会中的许多问题，这些问题不适合复杂的基于模型的推理和决策，现在将变得易于处理。该项目中开发的算法将作为开源软件发布，以便广大的研究人员和从业人员可以将其应用于一系列科学和工程问题。此外，在这个项目中开发的替代方法正在被纳入德克萨斯大学奥斯汀分校教授的一门关于反问题的流行研究生课程。近年来，神经算子由于其易于实现、对不同设置的适应性以及似乎能够减轻维数灾难而变得流行。重要的最近的研究试图建立“通用近似”的性质，这些代理人的各种类别的地图。虽然理论表明，神经运算符原则上可以达到任意精度，但在实践中实现这一点仍然是一个重大挑战。其原因包括生成足够的训练数据的巨大成本，以及统计抽样误差，近似误差和训练问题的非凸性之间的混淆关系。通常，相对于高保真PDE求解器，神经运算符只能达到1-2位数的精度，进一步降低精度的希望很小。另一方面，高保真度PDE模型（特别是守恒和平衡定律）通常具有非常高的置信度，并且由于PDE解对输入中的小扰动的敏感性，高精度是必要的。神经运算符的适度精度通常不足以满足关键系统的推理、控制和决策的需求。该项目正在开发神经算子和高保真PDE模型的混合体，以实现每个模型的最佳功能，通过PDE残差保持精度，并通过使用神经算子作为预处理器保持速度。该项目的目标是偏微分方程的线性和非线性参数神经预处理器，以及Metropolized Langevin方法的神经预处理器，以加速贝叶斯逆问题的解决方案。使用神经运算符作为预处理器的另一个优点是它们可以很好地映射到GPU架构上。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Many physical systems across all areas of science, engineering, medicine, and defense are modeled with high accuracy by partial differential equations (PDEs) and solved on advanced computing systems. Often the ultimate goal is to repeatedly solve the PDEs to explore parameter uncertainties. Settings in which this arises are inverse problems (inferring uncertain parameters of a model from data), optimal experimental design (determining the optimal data acquisition to learn the most about the model), optimal design (finding the optimal configuration of a system to maximize performance), and optimal control (determining the optimal operation of a system to achieve a desired behavior). These problems are often characterized by high dimensional uncertain parameter spaces, since the parameters typically represent initial conditions, boundary conditions, material properties, or source terms and vary in space and/or time. As a result, the PDEs often have to be solved thousands or even millions of times to adequately represent uncertainties in the parameters. When the systems that are modeled involve coupled multiple physics or behavior occurring on multiple space and time scales, repeated solution of the PDE models becomes prohibitive, even on the latest supercomputers. The development of deep neural networks in recent years shows promise in overcoming the intractability of repeated solution of the PDE models, by learning the relationships between the input parameters and the outputs of interest (e.g., temperature, velocity, pressure, stress, electric field, magnetic field, chemical species). Once trained on PDE solution data, the networks can evaluate the outputs for any given inputs in milliseconds, compared to hours or days to solve the PDE models themselves. However, despite much progress in the development of these so-called neural network surrogates, they typically deliver just 1-2 digits of accuracy, which is not sufficient to replace the PDE solver. Instead, this project is developing hybrids of neural network surrogates and PDE models that combine the best properties of each: the accuracy of the PDEs with the speed of the neural networks. The impact is that many problems in technology, health, the environment, and society that were not amenable to complex model-based inference and decision making will now become tractable. The algorithms developed in this project are being released as open-source software so that a broad community of researchers and practitioners can apply them to a spectrum of scientific and engineering problems. In addition, the surrogate methods developed in this project are being incorporated into a popular graduate course on inverse problems taught at University of Texas, Austin.Neural network approximations of high fidelity PDE solutions, i.e., neural operators, have gained popularity in recent years due to their ease of implementation, adaptability to varied settings, and seeming ability to mitigate the curse of dimensionality. Significant recent research has attempted to establish "universal approximation" properties of these surrogates for various classes of maps. While theory suggests that neural operators can in principle achieve arbitrary accuracy, realizing this in practice remains a significant challenge. The reasons for this include the enormous costs of generating sufficient training data, and confounding relations between statistical sampling errors, approximation errors, and nonconvexity of the training problem. Often neural operators can achieve just 1-2 digits of accuracy relative to high fidelity PDE solvers, with little hope of further reducing this accuracy. On the other hand, high fidelity PDE models (particularly conservation and balance laws) are often known with very high confidence and high precision is necessary due to sensitivity of PDE solutions to small perturbations in the inputs. The modest accuracies of neural operators are often insufficient for the demands of inference, control, and decision making for critical systems. This project is developing hybrids of neural operators and high fidelity PDE models to realize the best features of each, by retaining accuracy via the PDE residual and speed via use of the neural operator as a preconditioner. The project targets linear and nonlinear parametric neural preconditioners for PDEs, and neural preconditioners for Metropolized Langevin methods to accelerate the solution of Bayesian inverse problems. A further advantage of using neural operators as preconditioners is that they map well onto GPU architectures.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.