We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.SY

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Systems and Control

Title: Analysis of gradient descent methods with non-diminishing, bounded errors

Abstract: The main aim of this paper is to provide an analysis of gradient descent (GD) algorithms with gradient errors that do not necessarily vanish, asymptotically. In particular, sufficient conditions are presented for both stability (almost sure boundedness of the iterates) and convergence of GD with bounded, (possibly) non-diminishing gradient errors. In addition to ensuring stability, such an algorithm is shown to converge to a small neighborhood of the minimum set, which depends on the gradient errors. It is worth noting that the main result of this paper can be used to show that GD with asymptotically vanishing errors indeed converges to the minimum set. The results presented herein are not only more general when compared to previous results, but our analysis of GD with errors is new to the literature to the best of our knowledge. Our work extends the contributions of Mangasarian & Solodov, Bertsekas & Tsitsiklis and Tadic & Doucet. Using our framework, a simple yet effective implementation of GD using simultaneous perturbation stochastic approximations (SP SA), with constant sensitivity parameters, is presented. Another important improvement over many previous results is that there are no `additional' restrictions imposed on the step-sizes. In machine learning applications where step-sizes are related to learning rates, our assumptions, unlike those of other papers, do not affect these learning rates. Finally, we present experimental results to validate our theory.
Comments: arXiv admin note: text overlap with arXiv:1502.01953, IEEE Transactions on Automatic Control, 2017
Subjects: Systems and Control (eess.SY); Machine Learning (stat.ML)
MSC classes: 93E15, 93E35
Cite as: arXiv:1604.00151 [cs.SY]
  (or arXiv:1604.00151v3 [cs.SY] for this version)

Submission history

From: Arunselvan Ramaswamy [view email]
[v1] Fri, 1 Apr 2016 07:03:46 GMT (17kb)
[v2] Tue, 27 Sep 2016 14:36:07 GMT (27kb,D)
[v3] Mon, 18 Sep 2017 08:56:56 GMT (30kb,D)

Link back to: arXiv, form interface, contact.