Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studies
(Submitted on 13 Oct 2021 (this version), latest version 19 Oct 2021 (v2))
Abstract: We present and review an algorithmic and theoretical framework for improving neural network architecture design via momentum. As case studies, we consider how momentum can improve the architecture design for recurrent neural networks (RNNs), neural ordinary differential equations (ODEs), and transformers. We show that integrating momentum into neural network architectures has several remarkable theoretical and empirical benefits, including 1) integrating momentum into RNNs and neural ODEs can overcome the vanishing gradient issues in training RNNs and neural ODEs, resulting in effective learning long-term dependencies. 2) momentum in neural ODEs can reduce the stiffness of the ODE dynamics, which significantly enhances the computational efficiency in training and testing. 3) momentum can improve the efficiency and accuracy of transformers.
Submission history
From: Bao Wang [view email][v1] Wed, 13 Oct 2021 21:11:04 GMT (5267kb,D)
[v2] Tue, 19 Oct 2021 02:55:21 GMT (5267kb,D)
Link back to: arXiv, form interface, contact.