Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Scaling down Deep Learning
(Submitted on 29 Nov 2020 (v1), last revised 4 Dec 2020 (this version, v3))
Abstract: Though deep learning models have taken on commercial and political relevance, many aspects of their training and operation remain poorly understood. This has sparked interest in "science of deep learning" projects, many of which are run at scale and require enormous amounts of time, money, and electricity. But how much of this research really needs to occur at scale? In this paper, we introduce MNIST-1D: a minimalist, low-memory, and low-compute alternative to classic deep learning benchmarks. The training examples are 20 times smaller than MNIST examples yet they differentiate more clearly between linear, nonlinear, and convolutional models which attain 32, 68, and 94% accuracy respectively (these models obtain 94, 99+, and 99+% on MNIST). Then we present example use cases which include measuring the spatial inductive biases of lottery tickets, observing deep double descent, and metalearning an activation function.
Submission history
From: Sam Greydanus [view email][v1] Sun, 29 Nov 2020 20:08:37 GMT (2451kb,D)
[v2] Tue, 1 Dec 2020 22:09:02 GMT (2451kb,D)
[v3] Fri, 4 Dec 2020 20:09:44 GMT (3268kb,D)
Link back to: arXiv, form interface, contact.