Current browse context:
math.OC
Change to browse by:
References & Citations
Mathematics > Optimization and Control
Title: A Decentralized Parallel Algorithm for Training Generative Adversarial Nets
(Submitted on 28 Oct 2019 (v1), last revised 19 Oct 2020 (this version, v6))
Abstract: Generative Adversarial Networks (GANs) are a powerful class of generative models in the deep learning community. Current practice on large-scale GAN training utilizes large models and distributed large-batch training strategies, and is implemented on deep learning frameworks (e.g., TensorFlow, PyTorch, etc.) designed in a centralized manner. In the centralized network topology, every worker needs to either directly communicate with the central node or indirectly communicate with all other workers in every iteration. However, when the network bandwidth is low or network latency is high, the performance would be significantly degraded. Despite recent progress on decentralized algorithms for training deep neural networks, it remains unclear whether it is possible to train GANs in a decentralized manner. The main difficulty lies at handling the nonconvex-nonconcave min-max optimization and the decentralized communication simultaneously. In this paper, we address this difficulty by designing the \textbf{first gradient-based decentralized parallel algorithm} which allows workers to have multiple rounds of communications in one iteration and to update the discriminator and generator simultaneously, and this design makes it amenable for the convergence analysis of the proposed decentralized algorithm. Theoretically, our proposed decentralized algorithm is able to solve a class of non-convex non-concave min-max problems with provable non-asymptotic convergence to first-order stationary point. Experimental results on GANs demonstrate the effectiveness of the proposed algorithm.
Submission history
From: Mingrui Liu [view email][v1] Mon, 28 Oct 2019 22:51:45 GMT (87kb,D)
[v2] Wed, 30 Oct 2019 20:38:02 GMT (87kb,D)
[v3] Mon, 17 Feb 2020 08:12:30 GMT (31947kb,D)
[v4] Tue, 18 Feb 2020 03:00:10 GMT (5420kb,D)
[v5] Fri, 26 Jun 2020 06:34:49 GMT (2574kb,D)
[v6] Mon, 19 Oct 2020 20:51:45 GMT (2574kb,D)
Link back to: arXiv, form interface, contact.