We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

math.OC

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Neural and Evolutionary Computing

Title: Efficient Approximations of the Fisher Matrix in Neural Networks using Kronecker Product Singular Value Decomposition

Authors: Abdoulaye Koroko (IFPEN), Ani Anciaux-Sedrakian (IFPEN), Ibtihel Gharbia (IFPEN), Valérie Garès (IRMAR), Mounir Haddou (IRMAR), Quang Huy Tran (IFPEN)
Abstract: Several studies have shown the ability of natural gradient descent to minimize the objective function more efficiently than ordinary gradient descent based methods. However, the bottleneck of this approach for training deep neural networks lies in the prohibitive cost of solving a large dense linear system corresponding to the Fisher Information Matrix (FIM) at each iteration. This has motivated various approximations of either the exact FIM or the empirical one. The most sophisticated of these is KFAC, which involves a Kronecker-factored block diagonal approximation of the FIM. With only a slight additional cost, a few improvements of KFAC from the standpoint of accuracy are proposed. The common feature of the four novel methods is that they rely on a direct minimization problem, the solution of which can be computed via the Kronecker product singular value decomposition technique. Experimental results on the three standard deep auto-encoder benchmarks showed that they provide more accurate approximations to the FIM. Furthermore, they outperform KFAC and state-of-the-art first-order methods in terms of optimization speed.
Subjects: Neural and Evolutionary Computing (cs.NE); Optimization and Control (math.OC); Machine Learning (stat.ML)
Cite as: arXiv:2201.10285 [cs.NE]
  (or arXiv:2201.10285v5 [cs.NE] for this version)

Submission history

From: Abdoulaye Koroko [view email]
[v1] Tue, 25 Jan 2022 12:56:17 GMT (451kb,D)
[v2] Wed, 2 Feb 2022 14:45:16 GMT (459kb,D)
[v3] Mon, 21 Feb 2022 14:31:26 GMT (434kb,D)
[v4] Fri, 18 Mar 2022 09:07:55 GMT (434kb,D)
[v5] Tue, 12 Apr 2022 08:07:39 GMT (435kb,D)

Link back to: arXiv, form interface, contact.