We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Convergence and Implicit Regularization Properties of Gradient Descent for Deep Residual Networks

Abstract: We prove linear convergence of gradient descent to a global optimum for the training of deep residual networks with constant layer width and smooth activation function. We show that if the trained weights, as a function of the layer index, admit a scaling limit as the depth increases, then the limit has finite $p-$variation with $p=2$. Proofs are based on non-asymptotic estimates for the loss function and for norms of the network weights along the gradient descent path. We illustrate the relevance of our theoretical results to practical settings using detailed numerical experiments on supervised learning problems.
Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC)
MSC classes: 65Kxx, 62M45, 68Q32, 68Txx
Cite as: arXiv:2204.07261 [cs.LG]
  (or arXiv:2204.07261v4 [cs.LG] for this version)

Submission history

From: Alain Rossier [view email]
[v1] Thu, 14 Apr 2022 22:50:28 GMT (934kb,D)
[v2] Tue, 3 May 2022 19:16:34 GMT (467kb,D)
[v3] Mon, 23 Jan 2023 15:36:30 GMT (469kb,D)
[v4] Wed, 25 Jan 2023 16:51:52 GMT (469kb,D)

Link back to: arXiv, form interface, contact.