Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Implicit Regularization via Neural Feature Alignment
(Submitted on 3 Aug 2020 (v1), last revised 17 Mar 2021 (this version, v3))
Abstract: We approach the problem of implicit regularization in deep learning from a geometrical viewpoint. We highlight a regularization effect induced by a dynamical alignment of the neural tangent features introduced by Jacot et al, along a small number of task-relevant directions. This can be interpreted as a combined mechanism of feature selection and compression. By extrapolating a new analysis of Rademacher complexity bounds for linear models, we motivate and study a heuristic complexity measure that captures this phenomenon, in terms of sequences of tangent kernel classes along optimization paths.
Submission history
From: Aristide Baratin [view email][v1] Mon, 3 Aug 2020 15:18:07 GMT (10645kb,D)
[v2] Wed, 28 Oct 2020 17:04:22 GMT (25341kb,D)
[v3] Wed, 17 Mar 2021 00:57:56 GMT (25666kb,D)
Link back to: arXiv, form interface, contact.