We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Local Stochastic Gradient Descent Ascent: Convergence Analysis and Communication Efficiency

Abstract: Local SGD is a promising approach to overcome the communication overhead in distributed learning by reducing the synchronization frequency among worker nodes. Despite the recent theoretical advances of local SGD in empirical risk minimization, the efficiency of its counterpart in minimax optimization remains unexplored. Motivated by large scale minimax learning problems, such as adversarial robust learning and training generative adversarial networks (GANs), we propose local Stochastic Gradient Descent Ascent (local SGDA), where the primal and dual variables can be trained locally and averaged periodically to significantly reduce the number of communications. We show that local SGDA can provably optimize distributed minimax problems in both homogeneous and heterogeneous data with reduced number of communications and establish convergence rates under strongly-convex-strongly-concave and nonconvex-strongly-concave settings. In addition, we propose a novel variant local SGDA+, to solve nonconvex-nonconcave problems. We give corroborating empirical evidence on different distributed minimax problems.
Comments: This paper has been accepted to AISTATS 2021
Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC)
Cite as: arXiv:2102.13152 [cs.LG]
  (or arXiv:2102.13152v1 [cs.LG] for this version)

Submission history

From: Yuyang Deng [view email]
[v1] Thu, 25 Feb 2021 20:15:18 GMT (1921kb,D)

Link back to: arXiv, form interface, contact.