Current browse context:
stat
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure
(Submitted on 4 Jun 2021 (v1), last revised 10 Feb 2022 (this version, v2))
Abstract: Curvature in form of the Hessian or its generalized Gauss-Newton (GGN) approximation is valuable for algorithms that rely on a local model for the loss to train, compress, or explain deep networks. Existing methods based on implicit multiplication via automatic differentiation or Kronecker-factored block diagonal approximations do not consider noise in the mini-batch. We present ViViT, a curvature model that leverages the GGN's low-rank structure without further approximations. It allows for efficient computation of eigenvalues, eigenvectors, as well as per-sample first- and second-order directional derivatives. The representation is computed in parallel with gradients in one backward pass and offers a fine-grained cost-accuracy trade-off, which allows it to scale. We demonstrate this by conducting performance benchmarks and substantiate ViViT's usefulness by studying the impact of noise on the GGN's structural properties during neural network training.
Submission history
From: Felix Dangel [view email][v1] Fri, 4 Jun 2021 17:37:47 GMT (11827kb,D)
[v2] Thu, 10 Feb 2022 17:06:24 GMT (13027kb,D)
Link back to: arXiv, form interface, contact.