GDA-AM: On the Effectiveness of Solving Min-Imax Optimization via Anderson Mixing
GDA-AM: On the Effectiveness of Solving Min-Imax Optimization via Anderson Mixing
复制标题
DOI：
--
复制
发表时间：
2022
期刊：
影响因子：
--
通讯作者：
Huan He;Shifan Zhao;Yuanzhe Xi;Joyce C. Ho;Y. Saad
中科院分区：
文献类型：
--
作者：
Huan He;Shifan Zhao;Yuanzhe Xi;Joyce C. Ho;Y. Saad
许多现代机器学习算法，如生成对抗网络（GAN）和对抗训练，都可以用极大极小优化来表示。梯度下降上升（GDA）是最常用的算法，由于其简单。然而，GDA可以收敛到非最优极大极小点。我们提出了一个新的极大极小优化框架，GDA-AM，认为GDA动态作为一个固定点迭代，并解决它使用安德森混合收敛到本地极大极小。它解决了同步GDA的发散问题，加速了交替GDA的收敛。在理论上证明了该算法在较弱的条件下对双线性问题具有全局收敛性。我们还通过经验表明，GDA-AM解决了各种极大极小问题，并改善了几个数据集上的GAN训练。
Many modern machine learning algorithms such as generative adversarial networks (GANs) and adversarial training can be formulated as minimax optimization. Gradient descent ascent (GDA) is the most commonly used algorithm due to its simplicity. However, GDA can converge to non-optimal minimax points. We propose a new minimax optimization framework, GDA-AM, that views the GDA dynamics as a fixed-point iteration and solves it using Anderson Mixing to converge to the local minimax. It addresses the diverging issue of simultaneous GDA and accelerates the convergence of alternating GDA. We show theoretically that the algorithm can achieve global convergence for bilinear problems under mild conditions. We also empirically show that GDA-AM solves a variety of minimax problems and improves GAN training on several datasets.