Statistical and Computational Foundations of Deep Generative Models

复杂的数据每天都在科学和工程的各个领域不断产生，从照片或新闻文章到生物学或宇宙学实验。为了从这些材料流中提取有意义的信息，有必要构建适当的统计模型，忠实地表示每种数据模式。事实上，这样的统计模型对于评估数据分析方法对未来事件的预期性能是至关重要的，并且形成了称为“逆问题”的几个数据处理管道的关键组成部分。例如，从图像中去除噪声和缺陷，或预测最可能折叠的蛋白质都是逆问题的实例，其核心是需要期望输出的忠实统计模型。本项目的主要目标是推进基于神经网络的统计模型的理论基础。这类模型提供了比传统统计建模更大的灵活性，但结果是更难分析和操作。研究人员将涵盖机器学习，概率论，统计学和数学物理的广泛背景；他们结合的专业知识将产生指导原则，将神经网络结合到理论上健全的统计模型中，以及具有统计保证的新算法。研究成果将直接应用于宇宙学、气候模型、化学、信号处理等科学和工程领域的广泛问题，并将与教育课程紧密结合。深度学习（DL）在科学和工程领域的成功表明，深度神经网络（DNN）是复杂高维数据的有效函数近似模型，但这种能力的原因仍然知之甚少。为了在这个问题上取得进展，本项目侧重于生成概率建模。理解它们的内部工作原理对于解释深度学习在典型问题实例上的成功至关重要，而不是最坏情况（过于悲观）或非结构化（过于简单）的数据分布。此外，概率模型是许多科学学科中使用的计算工具的核心，但它们通常依赖于领域专业知识，使它们无法有效地随维度扩展。这个项目提出了一个统一的观点，同时解决近似，估计和优化方面的生成建模。具体来说，它涵盖了Boltzmann-Gibbs分布给出的显式建模和基于传输的模型（生成对抗网络，规范化流）给出的隐式建模。当使用dnn作为函数逼近时，它将从这些模型中建立学习和采样的保证。该项目将依赖于计算科学中开发的重要采样方法（如副本交换和热力学集成），并将其升级为与深度神经网络一起运行。它还将推导出结合隐式和显式生成建模的新算法。最后，它将利用物理先验，如对称性和多尺度结构，并评估它们在具有挑战性的领域，如分子预测、湍流、统计力学和探索强化学习方面的好处。调查人员综合了所有这些领域的专业知识，使他们完全有资格执行该项目。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Complex data are continuously generated across all areas of science and engineering on a daily basis, from photographs or news articles to biological or cosmological experiments. In order to extract meaningful information out of this stream of material, it is necessary to build appropriate statistical models that faithfully represent each data modality. Indeed, such statistical models are critical to assess the expected performance of data analysis methods on future events, and form a key component of several data processing pipelines called `inverse problems’. For example, removing noise and defects from an image, or predicting the most likely folding of a protein are instances of inverse problems that at their core require a faithful statistical model of the desired output. The main goal of this project is to advance the theoretical foundations of statistical models based on neural networks. Such classes of models provide greater flexibility than traditional statistical modeling, but as a result are harder to analyze and manipulate. The investigators will cover a wide background in machine learning, probability, statistics, and mathematical physics; their combined expertise will result in guiding principles to combine neural networks into a theoretically sound statistical modeling, as well as novel algorithms with statistical guarantees. The research outcomes will be directly applicable to a wide range of problems in science and engineering, ranging from cosmology, climate modeling, chemistry, and signal processing, and they will be tightly integrated into educational courses. The success of deep learning (DL) across science and engineering suggests that Deep Neural Networks (DNN) are effective function approximation models for complex high-dimensional data, yet the reasons for such capability are still poorly understood. To make headway on this problem, this project focuses on generative probabilistic modeling. Understanding their inner-workings is essential to explaining the success of DL on typical problem instances, as opposed to worst-case (too pessimistic) or unstructured (too simplistic) data distributions. Additionally, probabilistic models are at the core of computational tools used in many scientific disciplines, yet they often rely on domain expertise preventing them to scale efficiently with dimension. This project puts forward a unified view on generative modeling that simultaneously addresses approximation, estimation, and optimization aspects. Specifically, it covers both explicit modeling, given by Boltzmann-Gibbs distributions, and implicit modeling, given by Transport-based models (Generative Adversarial Networks, Normalizing Flows). It will establish guarantees of learning and sampling from these models when using DNNs as function approximation. This project will rely on methods for importance-sampling developed in computational sciences (such as Replica Exchange and Thermodynamic Integration) and upgrade them to operate alongside DNNs. It will also derive novel algorithms that combine implicit with explicit generative modeling. Finally, it will exploit physical priors such as symmetries and multiscale structure, and assess their benefits on challenging domains such as molecular prediction, turbulence, statistical mechanics, and exploration for reinforcement learning. The investigators have combined expertise in all these areas, making them well qualified to carry out the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.