Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networks
(Submitted on 24 Feb 2020 (v1), last revised 15 Aug 2020 (this version, v2))
Abstract: We develop exact representations of training two-layer neural networks with rectified linear units (ReLUs) in terms of a single convex program with number of variables polynomial in the number of training samples and the number of hidden neurons. Our theory utilizes semi-infinite duality and minimum norm regularization. We show that ReLU networks trained with standard weight decay are equivalent to block $\ell_1$ penalized convex models. Moreover, we show that certain standard convolutional linear networks are equivalent semi-definite programs which can be simplified to $\ell_1$ regularized linear models in a polynomial sized discrete Fourier feature space.
Submission history
From: Tolga Ergen [view email][v1] Mon, 24 Feb 2020 21:32:41 GMT (3182kb)
[v2] Sat, 15 Aug 2020 05:26:03 GMT (8839kb,D)
Link back to: arXiv, form interface, contact.