We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Random Weight Factorization Improves the Training of Continuous Neural Representations

Abstract: Continuous neural representations have recently emerged as a powerful and flexible alternative to classical discretized representations of signals. However, training them to capture fine details in multi-scale signals is difficult and computationally expensive. Here we propose random weight factorization as a simple drop-in replacement for parameterizing and initializing conventional linear layers in coordinate-based multi-layer perceptrons (MLPs) that significantly accelerates and improves their training. We show how this factorization alters the underlying loss landscape and effectively enables each neuron in the network to learn using its own self-adaptive learning rate. This not only helps with mitigating spectral bias, but also allows networks to quickly recover from poor initializations and reach better local minima. We demonstrate how random weight factorization can be leveraged to improve the training of neural representations on a variety of tasks, including image regression, shape representation, computed tomography, inverse rendering, solving partial differential equations, and learning operators between function spaces.
Comments: 33 pages, 21 figures, 12 tables
Subjects: Machine Learning (cs.LG)
Cite as: arXiv:2210.01274 [cs.LG]
  (or arXiv:2210.01274v2 [cs.LG] for this version)

Submission history

From: Sifan Wang [view email]
[v1] Mon, 3 Oct 2022 23:48:48 GMT (9204kb,D)
[v2] Wed, 5 Oct 2022 13:12:28 GMT (9206kb,D)

Link back to: arXiv, form interface, contact.