Understanding Overparameterization in Generative Adversarial Networks
Understanding Overparameterization in Generative Adversarial Networks
复制标题
DOI：
--
复制
发表时间：
2021-04
期刊：
影响因子：
--
通讯作者：
Y. Balaji;M. Sajedi;N. Kalibhat;Mucong Ding;Dominik Stöger;M. Soltanolkotabi;S. Feizi
中科院分区：
文献类型：
--
作者：
Y. Balaji;M. Sajedi;N. Kalibhat;Mucong Ding;Dominik Stöger;M. Soltanolkotabi;S. Feizi
一大类无监督的深度学习方法，如生成性对抗网络(GANS)，涉及训练过度参数化的模型，其中模型的参数数量超过一定的阈值。有监督学习领域的大量工作已经证明了模型过度参数在梯度下降(GD)收敛到全局最优解中的重要性。相比之下，无监督设置和GAN尤其涉及通常使用梯度下降/上升(GDA)训练的非凸凹极小最大优化问题。在非凸凹问题中，模型超参数在GDA收敛到全局鞍点中的作用和好处还远未被了解。在这项工作中，我们从理论和经验两个方面全面分析了模型超参数化在广义遗传算法中的重要性。我们从理论上证明了在具有$1层神经网络生成器和线性鉴别器的过参数GaN模型中，GDA收敛到底层非凸凹极小-极大问题的全局鞍点。据我们所知，这是在这种情况下GDA全球趋同的第一个结果。我们的理论基于一个更一般的结果，它适用于服从某些假设(包括更深层次的生成器和随机特征鉴别器)的更广泛类型的非线性生成器和鉴别器。我们还通过在CIFAR-10和Celeb-A数据集上的几个大规模实验，对GANS中模型过度参数化的作用进行了实证研究。我们的实验表明，过度参数化提高了不同模型架构和数据集上生成的样本的质量。值得注意的是，我们观察到过度参数化会导致GDA的整体收敛行为更快、更稳定。
A broad class of unsupervised deep learning methods such as Generative Adversarial Networks (GANs) involve training of overparameterized models where the number of parameters of the model exceeds a certain threshold. A large body of work in supervised learning have shown the importance of model overparameterization in the convergence of the gradient descent (GD) to globally optimal solutions. In contrast, the unsupervised setting and GANs in particular involve non-convex concave mini-max optimization problems that are often trained using Gradient Descent/Ascent (GDA). The role and benefits of model overparameterization in the convergence of GDA to a global saddle point in non-convex concave problems is far less understood. In this work, we present a comprehensive analysis of the importance of model overparameterization in GANs both theoretically and empirically. We theoretically show that in an overparameterized GAN model with a $1$-layer neural network generator and a linear discriminator, GDA converges to a global saddle point of the underlying non-convex concave min-max problem. To the best of our knowledge, this is the first result for global convergence of GDA in such settings. Our theory is based on a more general result that holds for a broader class of nonlinear generators and discriminators that obey certain assumptions (including deeper generators and random feature discriminators). We also empirically study the role of model overparameterization in GANs using several large-scale experiments on CIFAR-10 and Celeb-A datasets. Our experiments show that overparameterization improves the quality of generated samples across various model architectures and datasets. Remarkably, we observe that overparameterization leads to faster and more stable convergence behavior of GDA across the board.