Decentralized Parallel Algorithm for Training Generative Adversarial Nets
Decentralized Parallel Algorithm for Training Generative Adversarial Nets
复制标题
DOI：
--
复制
发表时间：
2019-10
期刊：
影响因子：
--
通讯作者：
Mingrui Liu;Youssef Mroueh;Wei Zhang;Xiaodong Cui;Jerret Ross;Tianbao Yang;Payel Das
中科院分区：
文献类型：
--
作者：
Mingrui Liu;Youssef Mroueh;Wei Zhang;Xiaodong Cui;Jerret Ross;Tianbao Yang;Payel Das
生成对抗网络（GANs）是深度学习领域中强大的一类生成模型。目前大规模GAN训练的实践\cite{brock2018large}利用大型模型和分布式大批量训练策略，并在集中设计的深度学习框架（如TensorFlow， PyTorch等）上实现。在集中式网络拓扑中，每个工作人员都需要与中心节点通信。但是，当网络带宽较低或网络延迟较高时，性能会明显下降。尽管最近在训练深度神经网络的去中心化算法方面取得了进展，但是否有可能以去中心化的方式训练gan仍不清楚。本文设计了一种求解一类具有可证明保证的非凸非凹最小-最大问题的分散算法。在gan上的实验结果证明了该算法的有效性。
Generative Adversarial Networks (GANs) are powerful class of generative models in the deep learning community. Current practice on large-scale GAN training \cite{brock2018large} utilizes large models and distributed large-batch training strategies, and is implemented on deep learning frameworks (e.g., TensorFlow, PyTorch, etc.) designed in a centralized manner. In the centralized network topology, every worker needs to communicate with the central node. However, when the network bandwidth is low or network latency is high, the performance would be significantly degraded. Despite recent progress on decentralized algorithms for training deep neural networks, it remains unclear whether it is possible to train GANs in a decentralized manner. In this paper, we design a decentralized algorithm for solving a class of non-convex non-concave min-max problem with provable guarantee. Experimental results on GANs demonstrate the effectiveness of the proposed algorithm.