AF: Medium: Collaborative Research: Theoretical Foundations of Deep Generative Models and High-Dimensional Distributions

AF：中：协作研究：深度生成模型和高维分布的理论基础

• 批准号: 1901292
• 负责人: Constantinos Daskalakis
• 金额: $ 60万
• 依托单位: Massachusetts Institute of Technology
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-07-01 至 2025-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1901292&HistoricalAwards=false
• 关键词: AF; Medium; Collaborative; Research; Theoretical

Current technology is driving our ability to collect, store and process data at an unprecedented scale. Ranging from image, audio and video to social-network, medical and biological datasets, modern applications require us to model and reason about complex data over extremely large domains. It is well-known, however, that this cannot be done in a rigorous manner unless simplifying assumptions can be made about how the data of interest are generated. Accordingly, a long line of investigation in Probability Theory, Statistical Physics, Information Theory and Machine Learning has been preoccupied with developing mathematical and algorithmic frameworks that allow for succinct representation and inference of high-dimensional distributions with simplifying structure. This project will go beyond the standard frameworks in these fields to advance the theoretical foundations of a research frontier that has recently emerged as a promising approach towards a more accurate modeling of high-dimensional data. In particular, this project will study the theoretical foundations of learning, testing and statistical inference of high-dimensional data that are generated by deep neural network-based generative models, developing mathematically rigorous quality guarantees, which is a big desideratum in the field of deep learning. On the practical front, this work has the potential to significantly improve the performance of image-reconstruction algorithms compared to state-of-the-art, and therefore to have significant impact on various applications of image reconstruction such as rapid magnetic resonance imaging (MRI).Since the introduction of deep neural network-based generative models, there have been numerous approaches for how to architect them, how to train them using samples from a distribution of interest, and how to use them for downstream inference tasks; these have delivered impressive practical results. On the other hand, there has also been a lot of debate around the quality of deep generative models that are trained via current techniques, and it has been recognized that there are significant challenges in optimizing, evaluating and scaling the dimensionality of deep generative models, as well as in using them for data recovery. This project develops three research thrusts targeting these challenges, namely: (i) developing better algorithms for training deep generative models, and for using these models as "regularizers" in signal-processing applications; (ii) developing statistical techniques for evaluating the quality of a deep generative model against the distribution whose samples it was trained on; (iii) proposing architectures and algorithms for scaling up the dimensionality of deep generating models while providing statistical accuracy guarantees. This work will rely on techniques from non-convex and combinatorial optimization, signal processing, game theory, high-dimensional statistics, and statistical physics, and build connections between these fields and deep learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

当前的技术正在以前所未有的规模推动我们收集、存储和处理数据的能力。从图像、音频和视频到社交网络、医疗和生物数据集，现代应用要求我们对极其庞大的领域中的复杂数据进行建模和推理。然而，众所周知，除非能够对感兴趣的数据是如何产生的作出简化假设，否则无法以严格的方式做到这一点。因此，在概率论、统计物理学、信息论和机器学习方面的长期研究一直专注于开发数学和算法框架，这些框架允许以简化的结构对高维分布进行简洁的表示和推断。该项目将超越这些领域的标准框架，推进一个研究前沿的理论基础，该前沿最近成为一种有前途的方法，可以更准确地建模高维数据。特别是，该项目将研究基于深度神经网络的生成模型生成的高维数据的学习，测试和统计推断的理论基础，开发数学上严格的质量保证，这是深度学习领域的一大需求。在实践方面，与最先进的技术相比，这项工作有可能显着提高图像重建算法的性能，因此对图像重建的各种应用产生重大影响，例如快速磁共振成像（MRI）。自从引入基于深度神经网络的生成模型以来，已经有许多方法来构建它们，如何使用来自感兴趣分布的样本来训练它们，以及如何将它们用于下游推理任务;这些都提供了令人印象深刻的实际结果。另一方面，围绕通过当前技术训练的深度生成模型的质量也存在很多争论，并且人们已经认识到，在优化，评估和扩展深度生成模型的维度以及使用它们进行数据恢复方面存在重大挑战。该项目针对这些挑战开发了三个研究方向，即：（i）开发更好的算法来训练深度生成模型，并将这些模型用作信号处理应用中的“正则化器”;（ii）开发统计技术，用于评估深度生成模型的质量，以对抗其训练样本的分布;（iii）提出架构和算法，用于扩大深度生成模型的维度，同时提供统计准确性保证。这项工作将依赖于非凸和组合优化、信号处理、博弈论、高维统计和统计物理等技术，并在这些领域与深度学习之间建立联系。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

STay-ON-the-Ridge: Guaranteed Convergence to Local Minimax Equilibrium in Nonconvex-Nonconcave Games

• DOI: 10.48550/arxiv.2210.09769
• 发表时间: 2022-10
• 作者: C. Daskalakis;Noah Golowich;Stratis Skoulakis;Manolis Zampetakis

GANs with Conditional Independence Graphs: On Subadditivity of Probability Divergences

具有条件独立图的 GAN：概率散度的次可加性

• 发表时间: 2021
• 期刊: the 24th International Conference on Artificial Intelligence and Statistics (AISTATS
• 作者: Ding, Mucong;Daskalakis, Constantinos;Feizi, Soheil
• 通讯作者: Feizi, Soheil

Online Learning and Solving Infinite Games with an ERM Oracle.

使用 ERM Oracle 在线学习和解决无限游戏。

• 发表时间: 2023
• 期刊: Proceedings of the 36th Annual Conference on Learning Theory (COLT
• 作者: Angelos Assos;Idan Attias;Yuval Dagan;Constantinos Daskalakis;Maxwell Fishelson
• 通讯作者: Maxwell Fishelson

Constant-Expansion Suffices for Compressed Sensing with Generative Priors

• 发表时间: 2020-06
• 期刊: ArXiv
• 作者: C. Daskalakis;Dhruv Rohatgi;Manolis Zampetakis

Near-optimal no-regret learning for correlated equilibria in multi-player general-sum games

多人一般和博弈中相关均衡的近乎最优无悔学习

• DOI: 10.1145/3519935.3520031
• 发表时间: 2022
• 期刊: STOC-21
• 作者: Anagnostides, Ioannis;Daskalakis, Constantinos;Farina, Gabriele;Fishelson, Maxwell;Golowich, Noah;Sandholm, Tuomas
• 通讯作者: Sandholm, Tuomas