We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studies

Abstract: We present and review an algorithmic and theoretical framework for improving neural network architecture design via momentum. As case studies, we consider how momentum can improve the architecture design for recurrent neural networks (RNNs), neural ordinary differential equations (ODEs), and transformers. We show that integrating momentum into neural network architectures has several remarkable theoretical and empirical benefits, including 1) integrating momentum into RNNs and neural ODEs can overcome the vanishing gradient issues in training RNNs and neural ODEs, resulting in effective learning long-term dependencies. 2) momentum in neural ODEs can reduce the stiffness of the ODE dynamics, which significantly enhances the computational efficiency in training and testing. 3) momentum can improve the efficiency and accuracy of transformers.
Comments: 40 pages, 15 figures. arXiv admin note: substantial text overlap with arXiv:2006.06919, arXiv:2110.04840
Subjects: Machine Learning (cs.LG); Dynamical Systems (math.DS); Numerical Analysis (math.NA)
MSC classes: 68Txx
Cite as: arXiv:2110.07034 [cs.LG]
  (or arXiv:2110.07034v2 [cs.LG] for this version)

Submission history

From: Bao Wang [view email]
[v1] Wed, 13 Oct 2021 21:11:04 GMT (5267kb,D)
[v2] Tue, 19 Oct 2021 02:55:21 GMT (5267kb,D)

Link back to: arXiv, form interface, contact.