Current browse context:
stat.ML
Change to browse by:
References & Citations
Statistics > Machine Learning
Title: Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity
(Submitted on 30 Jun 2021 (v1), last revised 31 Jan 2022 (this version, v2))
Abstract: The dynamics of Deep Linear Networks (DLNs) is dramatically affected by the variance $\sigma^2$ of the parameters at initialization $\theta_0$. For DLNs of width $w$, we show a phase transition w.r.t. the scaling $\gamma$ of the variance $\sigma^2=w^{-\gamma}$ as $w\to\infty$: for large variance ($\gamma<1$), $\theta_0$ is very close to a global minimum but far from any saddle point, and for small variance ($\gamma>1$), $\theta_0$ is close to a saddle point and far from any global minimum. While the first case corresponds to the well-studied NTK regime, the second case is less understood. This motivates the study of the case $\gamma \to +\infty$, where we conjecture a Saddle-to-Saddle dynamics: throughout training, gradient descent visits the neighborhoods of a sequence of saddles, each corresponding to linear maps of increasing rank, until reaching a sparse global minimum. We support this conjecture with a theorem for the dynamics between the first two saddles, as well as some numerical experiments.
Submission history
From: Arthur Jacot [view email][v1] Wed, 30 Jun 2021 09:34:05 GMT (715kb,D)
[v2] Mon, 31 Jan 2022 12:58:39 GMT (1550kb,D)
Link back to: arXiv, form interface, contact.