References & Citations
Mathematics > Optimization and Control
Title: Accelerated Primal-Dual Algorithm for Distributed Non-convex Optimization
(Submitted on 13 Aug 2021 (v1), last revised 13 Oct 2021 (this version, v3))
Abstract: This paper investigates accelerating the convergence of distributed optimization algorithms on non-convex problems. We propose a distributed primal-dual stochastic gradient descent~(SGD) equipped with "powerball" method to accelerate. We show that the proposed algorithm achieves the linear speedup convergence rate $\mathcal{O}(1/\sqrt{nT})$ for general smooth (possibly non-convex) cost functions. We demonstrate the efficiency of the algorithm through numerical experiments by training two-layer fully connected neural networks and convolutional neural networks on the MNIST dataset to compare with state-of-the-art distributed SGD algorithms and centralized SGD algorithms.
Submission history
From: Shengjun Zhang [view email][v1] Fri, 13 Aug 2021 03:59:44 GMT (1233kb)
[v2] Fri, 20 Aug 2021 02:18:54 GMT (1234kb)
[v3] Wed, 13 Oct 2021 21:53:28 GMT (1236kb)
Link back to: arXiv, form interface, contact.