Current browse context:
stat.ML
Change to browse by:
References & Citations
Statistics > Machine Learning
Title: Tractable structured natural gradient descent using local parameterizations
(Submitted on 15 Feb 2021 (v1), last revised 17 Jan 2022 (this version, v10))
Abstract: Natural-gradient descent (NGD) on structured parameter spaces (e.g., low-rank covariances) is computationally challenging due to difficult Fisher-matrix computations. We address this issue by using \emph{local-parameter coordinates} to obtain a flexible and efficient NGD method that works well for a wide-variety of structured parameterizations. We show four applications where our method (1) generalizes the exponential natural evolutionary strategy, (2) recovers existing Newton-like algorithms, (3) yields new structured second-order algorithms via matrix groups, and (4) gives new algorithms to learn covariances of Gaussian and Wishart-based distributions. We show results on a range of problems from deep learning, variational inference, and evolution strategies. Our work opens a new direction for scalable structured geometric methods.
Submission history
From: Wu Lin [view email][v1] Mon, 15 Feb 2021 09:09:20 GMT (1530kb,D)
[v2] Thu, 4 Mar 2021 10:00:09 GMT (1533kb,D)
[v3] Tue, 6 Apr 2021 09:36:48 GMT (2247kb,D)
[v4] Mon, 3 May 2021 05:48:07 GMT (2251kb,D)
[v5] Tue, 22 Jun 2021 22:31:08 GMT (3544kb,D)
[v6] Mon, 12 Jul 2021 10:54:59 GMT (3709kb,D)
[v7] Sun, 1 Aug 2021 07:35:43 GMT (3735kb,D)
[v8] Thu, 30 Sep 2021 20:33:39 GMT (3736kb,D)
[v9] Mon, 15 Nov 2021 17:12:52 GMT (3736kb,D)
[v10] Mon, 17 Jan 2022 17:09:53 GMT (3736kb,D)
Link back to: arXiv, form interface, contact.