Unifying Information- and Optimization-Theoretic Approaches for Modeling and Training Generative Adversarial Networks

现代机器学习(ML)的成功是由学习复杂预测模型所需的大量数据推动的。然而，缺乏公开可用的数据来开发和测试ML算法具有巨大的社会影响，包括对黑盒模型中算法偏差的不可证实的担忧。对于电网等关键系统来说，对公共数据集的需求更加强烈，在这些系统中，现实的数据集对于实时决策和稳健的长期规划都是必不可少的。合成数据承诺以安全和一致的方式开发ML算法；然而，开发有原则的方法来生成合成数据并保证了解数据分布仍然是一个悬而未决的问题。生成性对抗网络(GANS)已经成为生成合成数据的一种有效的深度学习方法。GANS涉及两个模块，在实践中被建模为深度神经网络：一个是合成样本的生成器，另一个是辨别器，它将输入的样本分类为真或假。两个模块的相对目标产生了一个最小-最大(MIN-MAX)博弈。尽管他们取得了成功，但由于一系列不稳定问题、不收敛和模式崩溃，GAN很难训练。教育部分培训研究生、博士后和本科生，特别是来自代表不足的少数群体(通过亚利桑那州立大学夏季本科生研究计划)的学生，涉及电气工程、计算机科学和统计学，以应对数据科学和机器学习中的新挑战。该项目开发了一个统一的信息和优化理论框架来应对这些挑战，利用信息论、优化和博弈论、贝叶斯方法和随机顺序搜索技术。通过恢复几种常用GaN的基于损耗函数的GaN可调框架，解决了消失梯度和GaN损失函数之间的联系。该项目以两种新颖的方式处理GAN优化问题：(I)建立一类一般的非凸函数极大极小问题解的存在性；(Ii)引入统一变分不等式(VI)框架来系统地解决确定性和随机VI问题，包括极小极大Gans问题。该项目通过训练具有贝叶斯先验的生成器和鉴别器参数的GAN来解决模式崩溃问题。这种方法的一个关键创新之处在于，使用反问题方法确定了潜在空间在模式崩溃中的作用。最后，该项目应用和评估了所提出的训练可调遗传算法的方法，并开发了一种随机序列搜索算法来确保训练的遗传算法的全局最优性。理论结果使用公共和专有数据集进行评估。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The success of modern machine learning (ML) is driven by copious amounts of data needed to learn complex predictive models. However, the lack of publicly available data to develop and test ML algorithms has large societal implications including unverifiable concerns of algorithmic bias in blackbox models. The need for public datasets is even stronger for critical systems such as the electric grid where realistic datasets are essential for both real-time decision-making and robust long-term planning. Synthetic data promises a secure and consistent way to develop ML algorithms; yet, developing principled methods to generate synthetic data with guarantees on learning the data distribution continues to be an open problem. Generative adversarial networks (GANs) have emerged as an effective deep learning approach for generating synthetic data. GANs involve two modules, modeled in practice as deep neural networks: a generator of synthetic samples, and a discriminator which classifies inputs to it as real or fake. The opposing goals of the two modules yields a minimum-maximum (min-max) game. Despite their success, GANs are difficult to train due to a range of instability problems, non-convergence, and mode collapse. The educational component trains graduate students, postdocs, and undergraduates, particularly from underrepresented minority groups (via an Arizona State University Summer Undergraduate Research Initiative) across electrical engineering, computer science, and statistics to emerging challenges in data science and machine learning.This project develops a unified information- and optimization-theoretic framework to address these challenges, leveraging information theory, optimization and game theory, Bayesian methods, and stochastic sequential search techniques. Connections between vanishing gradients and GAN loss functions is addressed via a loss function-based tunable framework for GANs that recovers several oft-used GANs. The project tackles GAN optimization problems in two novel ways: (i) establishing existence of solutions for a general class of nonconvex and functional min-max problems; and (ii) introducing a unifying variational inequality (VI) framework to systematically solve deterministic and stochastic VI problems, including min-max GANs. The project addresses mode collapse by training GANs with Bayesian priors on generator and discriminator parameters. A key novelty of this approach is in identifying the role of latent space on mode collapse using an inverse problem methodology. Finally, the project applies and evaluates the proposed approaches to training tunable GANs and develops a stochastic sequential search algorithm to assure global optimality of trained GANs. Theoretical results are evaluated using both public and proprietary datasets.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.