We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Towards Modeling and Resolving Singular Parameter Spaces using Stratifolds

Abstract: When analyzing parametric statistical models, a useful approach consists in modeling geometrically the parameter space. However, even for very simple and commonly used hierarchical models like statistical mixtures or stochastic deep neural networks, the smoothness assumption of manifolds is violated at singular points which exhibit non-smooth neighborhoods in the parameter space. These singular models have been analyzed in the context of learning dynamics, where singularities can act as attractors on the learning trajectory and, therefore, negatively influence the convergence speed of models. We propose a general approach to circumvent the problem arising from singularities by using stratifolds, a concept from algebraic topology, to formally model singular parameter spaces. We use the property that specific stratifolds are equipped with a resolution method to construct a smooth manifold approximation of the singular space. We empirically show that using (natural) gradient descent on the smooth manifold approximation instead of the singular space allows us to avoid the attractor behavior and therefore improve the convergence speed in learning.
Comments: A preliminary version of this work was presented at NeurIPS 2021 as a Spotlight in the 13th Annual Workshop on Optimization for Machine Learning (OPT2021)
Subjects: Machine Learning (cs.LG)
Cite as: arXiv:2112.03734 [cs.LG]
  (or arXiv:2112.03734v1 [cs.LG] for this version)

Submission history

From: Pascal Mattia Esser [view email]
[v1] Tue, 7 Dec 2021 14:42:45 GMT (363kb,D)

Link back to: arXiv, form interface, contact.