We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Algorithmic Regularization in Over-parameterized Matrix Sensing and Neural Networks with Quadratic Activations

Abstract: We show that the (stochastic) gradient descent algorithm provides an implicit regularization effect in the learning of over-parameterized matrix factorization models and one-hidden-layer neural networks with quadratic activations. Concretely, we show that given $\tilde{O}(dr^{2})$ random linear measurements of a rank $r$ positive semidefinite matrix $X^{\star}$, we can recover $X^{\star}$ by parameterizing it by $UU^\top$ with $U\in \mathbb{R}^{d\times d}$ and minimizing the squared loss, even if $r \ll d$. We prove that starting from a small initialization, gradient descent recovers $X^{\star}$ in $\tilde{O}(\sqrt{r})$ iterations approximately. The results solve the conjecture of Gunasekar et al.'17 under the restricted isometry property. The technique can be applied to analyzing neural networks with quadratic activations with some technical modifications.
Subjects: Machine Learning (cs.LG); Data Structures and Algorithms (cs.DS); Optimization and Control (math.OC); Machine Learning (stat.ML)
Cite as: arXiv:1712.09203 [cs.LG]
  (or arXiv:1712.09203v2 [cs.LG] for this version)

Submission history

From: Hongyang Zhang [view email]
[v1] Tue, 26 Dec 2017 08:04:43 GMT (397kb)
[v2] Wed, 21 Feb 2018 08:21:52 GMT (71kb)
[v3] Thu, 22 Feb 2018 03:50:08 GMT (71kb)
[v4] Mon, 19 Mar 2018 07:59:32 GMT (73kb)
[v5] Thu, 14 Feb 2019 00:24:05 GMT (109kb)

Link back to: arXiv, form interface, contact.