Current browse context:
stat.ML
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Neural networks learn to magnify areas near decision boundaries
(Submitted on 26 Jan 2023 (v1), last revised 14 Oct 2023 (this version, v3))
Abstract: In machine learning, there is a long history of trying to build neural networks that can learn from fewer example data by baking in strong geometric priors. However, it is not always clear a priori what geometric constraints are appropriate for a given task. Here, we consider the possibility that one can uncover useful geometric inductive biases by studying how training molds the Riemannian geometry induced by unconstrained neural network feature maps. We first show that at infinite width, neural networks with random parameters induce highly symmetric metrics on input space. This symmetry is broken by feature learning: networks trained to perform classification tasks learn to magnify local areas along decision boundaries. This holds in deep networks trained on high-dimensional image classification tasks, and even in self-supervised representation learning. These results begins to elucidate how training shapes the geometry induced by unconstrained neural network feature maps, laying the groundwork for an understanding of this richly nonlinear form of feature learning.
Submission history
From: Jacob Zavatone-Veth [view email][v1] Thu, 26 Jan 2023 19:43:16 GMT (20018kb,D)
[v2] Wed, 17 May 2023 17:07:01 GMT (18656kb,D)
[v3] Sat, 14 Oct 2023 22:52:45 GMT (22814kb,D)
Link back to: arXiv, form interface, contact.