We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cond-mat.dis-nn

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Condensed Matter > Disordered Systems and Neural Networks

Title: Soft Mode in the Dynamics of Over-realizable On-line Learning for Soft Committee Machines

Abstract: Over-parametrized deep neural networks trained by stochastic gradient descent are successful in performing many tasks of practical relevance. One aspect of over-parametrization is the possibility that the student network has a larger expressivity than the data generating process. In the context of a student-teacher scenario, this corresponds to the so-called over-realizable case, where the student network has a larger number of hidden units than the teacher. For on-line learning of a two-layer soft committee machine in the over-realizable case, we find that the approach to perfect learning occurs in a power-law fashion rather than exponentially as in the realizable case. All student nodes learn and replicate one of the teacher nodes if teacher and student outputs are suitably rescaled.
Comments: 5 pages, 5 figures
Subjects: Disordered Systems and Neural Networks (cond-mat.dis-nn); Machine Learning (cs.LG)
DOI: 10.1103/PhysRevE.105.L052302
Cite as: arXiv:2104.14546 [cond-mat.dis-nn]
  (or arXiv:2104.14546v1 [cond-mat.dis-nn] for this version)

Submission history

From: Bernd Rosenow [view email]
[v1] Thu, 29 Apr 2021 17:55:58 GMT (44kb,D)

Link back to: arXiv, form interface, contact.