Current browse context:
cond-mat.dis-nn
Change to browse by:
References & Citations
Condensed Matter > Disordered Systems and Neural Networks
Title: The star-shaped space of solutions of the spherical negative perceptron
(Submitted on 18 May 2023 (v1), last revised 5 Sep 2023 (this version, v2))
Abstract: Empirical studies on the landscape of neural networks have shown that low-energy configurations are often found in complex connected structures, where zero-energy paths between pairs of distant solutions can be constructed. Here we consider the spherical negative perceptron, a prototypical non-convex neural network model framed as a continuous constraint satisfaction problem. We introduce a general analytical method for computing energy barriers in the simplex with vertex configurations sampled from the equilibrium. We find that in the over-parameterized regime the solution manifold displays simple connectivity properties. There exists a large geodesically convex component that is attractive for a wide range of optimization dynamics. Inside this region we identify a subset of atypical high-margin solutions that are geodesically connected with most other solutions, giving rise to a star-shaped geometry. We analytically characterize the organization of the connected space of solutions and show numerical evidence of a transition, at larger constraint densities, where the aforementioned simple geodesic connectivity breaks down.
Submission history
From: Enrico Maria Malatesta [view email][v1] Thu, 18 May 2023 00:21:04 GMT (2620kb,D)
[v2] Tue, 5 Sep 2023 16:34:10 GMT (2730kb,D)
Link back to: arXiv, form interface, contact.