Current browse context:
stat.ML
Change to browse by:
References & Citations
Statistics > Machine Learning
Title: Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functions
(Submitted on 29 Sep 2022 (v1), last revised 23 Mar 2023 (this version, v4))
Abstract: We show that the representation cost of fully connected neural networks with homogeneous nonlinearities - which describes the implicit bias in function space of networks with $L_2$-regularization or with losses such as the cross-entropy - converges as the depth of the network goes to infinity to a notion of rank over nonlinear functions. We then inquire under which conditions the global minima of the loss recover the `true' rank of the data: we show that for too large depths the global minimum will be approximately rank 1 (underestimating the rank); we then argue that there is a range of depths which grows with the number of datapoints where the true rank is recovered. Finally, we discuss the effect of the rank of a classifier on the topology of the resulting class boundaries and show that autoencoders with optimal nonlinear rank are naturally denoising.
Submission history
From: Arthur Jacot [view email][v1] Thu, 29 Sep 2022 18:57:51 GMT (217kb,D)
[v2] Tue, 4 Oct 2022 19:05:19 GMT (217kb,D)
[v3] Thu, 13 Oct 2022 22:59:45 GMT (217kb,D)
[v4] Thu, 23 Mar 2023 18:14:14 GMT (133kb,D)
Link back to: arXiv, form interface, contact.