OAC Core: The Best of Both Worlds: Deep Neural Operators as Preconditioners for Physics-Based Forward and Inverse Problems

科学、工程、医学和国防领域的许多物理系统都是通过偏微分方程（PDEs）高精度建模的，并在先进的计算系统上求解。通常最终目标是反复求解偏微分方程以探索参数的不确定性。出现这种情况的设置是逆问题（从数据中推断模型的不确定参数），最佳实验设计（确定最佳数据采集以了解模型的大部分信息），最佳设计（找到系统的最佳配置以最大化性能）和最佳控制（确定系统的最佳操作以实现期望的行为）。这些问题通常以高维不确定参数空间为特征，因为参数通常代表初始条件、边界条件、材料属性或源项，并且在空间和/或时间上变化。因此，偏微分方程通常需要求解数千甚至数百万次才能充分表示参数中的不确定性。当所建模的系统涉及耦合的多个物理或行为发生在多个空间和时间尺度上时，即使在最新的超级计算机上，PDE模型的重复解决方案也变得令人望而却步。近年来，深度神经网络的发展表明，通过学习输入参数与感兴趣的输出（如温度、速度、压力、应力、电场、磁场、化学物质）之间的关系，有望克服PDE模型重复解的棘手性。一旦在PDE解决方案数据上进行训练，网络可以在毫秒内评估任何给定输入的输出，而解决PDE模型本身则需要数小时或数天。然而，尽管这些所谓的神经网络替代品的发展取得了很大进展，但它们通常只能提供1-2位数的精度，这不足以取代PDE求解器。相反，该项目正在开发神经网络替代品和PDE模型的混合模型，将两者的最佳特性结合起来：PDE的准确性和神经网络的速度。其影响是，技术、健康、环境和社会中的许多问题，过去不适合基于复杂模型的推理和决策，现在将变得容易处理。在这个项目中开发的算法将作为开源软件发布，以便广泛的研究人员和实践者社区可以将它们应用于一系列科学和工程问题。此外，在这个项目中开发的替代方法正在被纳入德克萨斯大学奥斯汀分校教授的一门流行的反问题研究生课程。高保真PDE解决方案的神经网络近似，即神经算子，近年来因其易于实现，对各种设置的适应性以及减轻维数诅咒的能力而受到欢迎。最近有重要的研究试图为各种类型的地图建立这些代理的“普遍近似”性质。虽然理论表明神经算子原则上可以达到任意精度，但在实践中实现这一点仍然是一个重大挑战。其原因包括生成足够训练数据的巨大成本，以及统计抽样误差、近似误差和训练问题的非凸性之间的混淆关系。通常，相对于高保真PDE求解器，神经算子只能达到1-2位的精度，并且几乎没有希望进一步降低这种精度。另一方面，高保真PDE模型（特别是守恒和平衡定律）通常以非常高的置信度已知，并且由于PDE解对输入中的小扰动的敏感性，高精度是必要的。神经算子的一般精度往往不足以满足关键系统的推理、控制和决策要求。该项目正在开发神经算子和高保真PDE模型的混合，通过使用PDE残差来保持精度，并通过使用神经算子作为前置条件来保持速度，从而实现每种模型的最佳特征。本课题针对偏微分方程的线性和非线性参数神经预调节器，以及metropolis Langevin方法的神经预调节器来加速贝叶斯反问题的求解。使用神经算子作为前置条件的另一个优势是，它们可以很好地映射到GPU架构上。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Many physical systems across all areas of science, engineering, medicine, and defense are modeled with high accuracy by partial differential equations (PDEs) and solved on advanced computing systems. Often the ultimate goal is to repeatedly solve the PDEs to explore parameter uncertainties. Settings in which this arises are inverse problems (inferring uncertain parameters of a model from data), optimal experimental design (determining the optimal data acquisition to learn the most about the model), optimal design (finding the optimal configuration of a system to maximize performance), and optimal control (determining the optimal operation of a system to achieve a desired behavior). These problems are often characterized by high dimensional uncertain parameter spaces, since the parameters typically represent initial conditions, boundary conditions, material properties, or source terms and vary in space and/or time. As a result, the PDEs often have to be solved thousands or even millions of times to adequately represent uncertainties in the parameters. When the systems that are modeled involve coupled multiple physics or behavior occurring on multiple space and time scales, repeated solution of the PDE models becomes prohibitive, even on the latest supercomputers. The development of deep neural networks in recent years shows promise in overcoming the intractability of repeated solution of the PDE models, by learning the relationships between the input parameters and the outputs of interest (e.g., temperature, velocity, pressure, stress, electric field, magnetic field, chemical species). Once trained on PDE solution data, the networks can evaluate the outputs for any given inputs in milliseconds, compared to hours or days to solve the PDE models themselves. However, despite much progress in the development of these so-called neural network surrogates, they typically deliver just 1-2 digits of accuracy, which is not sufficient to replace the PDE solver. Instead, this project is developing hybrids of neural network surrogates and PDE models that combine the best properties of each: the accuracy of the PDEs with the speed of the neural networks. The impact is that many problems in technology, health, the environment, and society that were not amenable to complex model-based inference and decision making will now become tractable. The algorithms developed in this project are being released as open-source software so that a broad community of researchers and practitioners can apply them to a spectrum of scientific and engineering problems. In addition, the surrogate methods developed in this project are being incorporated into a popular graduate course on inverse problems taught at University of Texas, Austin.Neural network approximations of high fidelity PDE solutions, i.e., neural operators, have gained popularity in recent years due to their ease of implementation, adaptability to varied settings, and seeming ability to mitigate the curse of dimensionality. Significant recent research has attempted to establish "universal approximation" properties of these surrogates for various classes of maps. While theory suggests that neural operators can in principle achieve arbitrary accuracy, realizing this in practice remains a significant challenge. The reasons for this include the enormous costs of generating sufficient training data, and confounding relations between statistical sampling errors, approximation errors, and nonconvexity of the training problem. Often neural operators can achieve just 1-2 digits of accuracy relative to high fidelity PDE solvers, with little hope of further reducing this accuracy. On the other hand, high fidelity PDE models (particularly conservation and balance laws) are often known with very high confidence and high precision is necessary due to sensitivity of PDE solutions to small perturbations in the inputs. The modest accuracies of neural operators are often insufficient for the demands of inference, control, and decision making for critical systems. This project is developing hybrids of neural operators and high fidelity PDE models to realize the best features of each, by retaining accuracy via the PDE residual and speed via use of the neural operator as a preconditioner. The project targets linear and nonlinear parametric neural preconditioners for PDEs, and neural preconditioners for Metropolized Langevin methods to accelerate the solution of Bayesian inverse problems. A further advantage of using neural operators as preconditioners is that they map well onto GPU architectures.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.