Current browse context:
math.OC
Change to browse by:
References & Citations
Mathematics > Optimization and Control
Title: Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers
(Submitted on 12 Oct 2019 (v1), last revised 15 Oct 2020 (this version, v5))
Abstract: We study the convergence of gradient flows related to learning deep linear neural networks (where the activation function is the identity map) from data. In this case, the composition of the network layers amounts to simply multiplying the weight matrices of all layers together, resulting in an overparameterized problem. The gradient flow with respect to these factors can be re-interpreted as a Riemannian gradient flow on the manifold of rank-$r$ matrices endowed with a suitable Riemannian metric. We show that the flow always converges to a critical point of the underlying functional. Moreover, we establish that, for almost all initializations, the flow converges to a global minimum on the manifold of rank $k$ matrices for some $k\leq r$.
Submission history
From: Holger Rauhut [view email][v1] Sat, 12 Oct 2019 06:51:27 GMT (566kb)
[v2] Mon, 25 Nov 2019 07:57:23 GMT (4232kb,D)
[v3] Wed, 19 Feb 2020 09:52:45 GMT (3336kb,D)
[v4] Mon, 24 Aug 2020 14:21:49 GMT (3363kb,D)
[v5] Thu, 15 Oct 2020 15:27:31 GMT (3363kb,D)
Link back to: arXiv, form interface, contact.