Existence and Estimation of Critical Batch Size for Training Generative Adversarial Networks with Two Time-Scale Update Rule
Existence and Estimation of Critical Batch Size for Training Generative Adversarial Networks with Two Time-Scale Update Rule
复制标题
DOI：
--
复制
发表时间：
2022-01
期刊：
影响因子：
--
通讯作者：
Naoki Sato;H. Iiduka
中科院分区：
文献类型：
--
作者：
Naoki Sato;H. Iiduka
先前的研究结果表明，使用不同学习率（例如不同的恒定速率或不同的衰减速率）的双时标更新规则（TTUR）在理论和实践中对于训练生成对抗网络（GAN）是有用的。此外，对于使用TTUR训练GAN来说，不仅学习率而且批量大小都很重要，它们都会影响训练所需的步骤数量。本文研究了批量大小与基于恒定学习率的TTUR训练GAN所需步骤数之间的关系。我们从理论上表明，对于一个TTUR与恒定的学习率，步骤的数量需要找到固定点的损失函数的两个？和发电机减少批量大小的增加，并存在一个临界批量大小最小化的随机一阶预言（SFO）的复杂性。然后，我们使用Fr'echet起始距离（FID）作为训练的性能测量，并提供数值结果，表明实现低FID分数所需的步骤数量随着批量大小的增加而减少，并且一旦批量大小超过测量的临界批量大小，SFO复杂性就会增加。此外，我们表明，测得的临界批量接近我们的理论结果估计的大小。
Previous results have shown that a two time-scale update rule (TTUR) using different learning rates, such as different constant rates or different decaying rates, is useful for training generative adversarial networks (GANs) in theory and in practice. Moreover, not only the learning rate but also the batch size is important for training GANs with TTURs and they both affect the number of steps needed for training. This paper studies the relationship between batch size and the number of steps needed for training GANs with TTURs based on constant learning rates. We theoretically show that, for a TTUR with constant learning rates, the number of steps needed to find stationary points of the loss functions of both the discriminator and generator decreases as the batch size increases and that there exists a critical batch size minimizing the stochastic first-order oracle (SFO) complexity. Then, we use the Fr'echet inception distance (FID) as the performance measure for training and provide numerical results indicating that the number of steps needed to achieve a low FID score decreases as the batch size increases and that the SFO complexity increases once the batch size exceeds the measured critical batch size. Moreover, we show that measured critical batch sizes are close to the sizes estimated from our theoretical results.