Disordered Systems and Neural Networks

New submissions

• New submissions
• Replacements

New submissions for Fri, 24 Mar 23

[1]  arXiv:2303.13506 [pdf, other]

Title: The Quantization Model of Neural Scaling

Authors: Eric J. Michaud, Ziming Liu, Uzay Girit, Max Tegmark

Comments: 24 pages, 15 figures

Subjects: Machine Learning (cs.LG); Disordered Systems and Neural Networks (cond-mat.dis-nn)

We propose the $\textit{Quantization Model}$ of neural scaling laws, explaining both the observed power law dropoff of loss with model and data size, and also the sudden emergence of new capabilities with scale. We derive this model from what we call the $\textit{Quantization Hypothesis}$, where learned network capabilities are quantized into discrete chunks ($\textit{quanta}$). We show that when quanta are learned in order of decreasing use frequency, then a power law in use frequencies explains observed power law scaling of loss. We validate this prediction on toy datasets, then study how scaling curves decompose for large language models. Using language model internals, we auto-discover diverse model capabilities (quanta) and find tentative evidence that the distribution over corresponding subproblems in the prediction of natural text is compatible with the power law predicted from the neural scaling exponent as predicted from our theory.

Replacements for Fri, 24 Mar 23

[2]  arXiv:2212.10768 (replaced) [pdf, ps, other]

Title: A stochastic method to compute the $L^2$ localisation landscape

Authors: Masataka Kakoi, Keith Slevin

Comments: 9 pages, 6 figures

Subjects: Disordered Systems and Neural Networks (cond-mat.dis-nn)

[3]  arXiv:2203.03060 (replaced) [pdf, other]

Title: Higher-order interactions shape collective dynamics differently in hypergraphs and simplicial complexes

Authors: Yuanzhao Zhang, Maxime Lucas, Federico Battiston

Comments: Published version. Y.Z. and M.L. contributed equally to this work. Code available at this https URL

Journal-ref: Nat. Commun. 14, 1605 (2023)

Subjects: Adaptation and Self-Organizing Systems (nlin.AO); Disordered Systems and Neural Networks (cond-mat.dis-nn); Systems and Control (eess.SY); Dynamical Systems (math.DS); Physics and Society (physics.soc-ph)

[4]  arXiv:2303.02010 (replaced) [pdf, other]

Title: Hayden-Preskill Recovery in Hamiltonian Systems

Authors: Yoshifumi Nakata, Masaki Tezuka

Comments: 7.2 pages, 4 figures, Supplemental Materials (13 pages, 10 figures)

Subjects: Strongly Correlated Electrons (cond-mat.str-el); Disordered Systems and Neural Networks (cond-mat.dis-nn); High Energy Physics - Theory (hep-th); Quantum Physics (quant-ph)

• New submissions
• Replacements