Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks
(Submitted on 30 Apr 2019 (v1), last revised 5 Dec 2019 (this version, v2))
Abstract: When optimizing over-parameterized models, such as deep neural networks, a large set of parameters can achieve zero training error. In such cases, the choice of the optimization algorithm and its respective hyper-parameters introduces biases that will lead to convergence to specific minimizers of the objective. Consequently, this choice can be considered as an implicit regularization for the training of over-parametrized models. In this work, we push this idea further by studying the discrete gradient dynamics of the training of a two-layer linear network with the least-squares loss. Using a time rescaling, we show that, with a vanishing initialization and a small enough step size, this dynamics sequentially learns the solutions of a reduced-rank regression with a gradually increasing rank.
Submission history
From: Gauthier Gidel [view email][v1] Tue, 30 Apr 2019 14:06:05 GMT (631kb,D)
[v2] Thu, 5 Dec 2019 17:02:21 GMT (1048kb,D)
Link back to: arXiv, form interface, contact.