Current browse context:
math.OC
Change to browse by:
References & Citations
Mathematics > Optimization and Control
Title: Fully Asynchronous Stochastic Coordinate Descent: A Tight Lower Bound on the Parallelism Achieving Linear Speedup
(Submitted on 8 Nov 2018 (v1), last revised 2 Aug 2020 (this version, v4))
Abstract: We seek tight bounds on the viable parallelism in asynchronous implementations of coordinate descent that achieves linear speedup. We focus on asynchronous coordinate descent (ACD) algorithms on convex functions which consist of the sum of a smooth convex part and a possibly non-smooth separable convex part.
We quantify the shortfall in progress compared to the standard sequential stochastic gradient descent. This leads to a simple yet tight analysis of the standard stochastic ACD in a partially asynchronous environment, generalizing and improving the bounds in prior work. We also give a considerably more involved analysis for general asynchronous environments in which the only constraint is that each update can overlap with at most q others. The new lower bound on the maximum degree of parallelism attaining linear speedup is tight and improves the best prior bound almost quadratically.
Submission history
From: Yun Kuen Cheung [view email][v1] Thu, 8 Nov 2018 03:52:51 GMT (76kb)
[v2] Sat, 25 May 2019 19:12:31 GMT (86kb)
[v3] Mon, 6 Jul 2020 21:04:01 GMT (87kb)
[v4] Sun, 2 Aug 2020 18:08:34 GMT (87kb)
Link back to: arXiv, form interface, contact.