We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.IT

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Information Theory

Title: Approximation speed of quantized vs. unquantized ReLU neural networks and beyond

Authors: Antoine Gonon (DANTE, ARIC), Nicolas Brisebarre (ARIC), Rémi Gribonval (DANTE), Elisa Riccietti (DANTE)
Abstract: We consider general approximation families encompassing ReLU neural networks. On the one hand, we introduce a new property, that we call $\infty$-encodability, which lays a framework that we use (i) to guarantee that ReLU networks can be uniformly quantized and still have approximation speeds comparable to unquantized ones, and (ii) to prove that ReLU networks share a common limitation with many other approximation families: the approximation speed of a set C is bounded from above by an encoding complexity of C (a complexity well-known for many C's). The property of $\infty$-encodability allows us to unify and generalize known results in which it was implicitly used. On the other hand, we give lower and upper bounds on the Lipschitz constant of the mapping that associates the weights of a network to the function they represent in L^p. It is given in terms of the width, the depth of the network and a bound on the weight's norm, and it is based on well-known upper bounds on the Lipschitz constants of the functions represented by ReLU networks. This allows us to recover known results, to establish new bounds on covering numbers, and to characterize the accuracy of naive uniform quantization of ReLU networks.
Subjects: Information Theory (cs.IT); Neural and Evolutionary Computing (cs.NE)
Cite as: arXiv:2205.11874 [cs.IT]
  (or arXiv:2205.11874v1 [cs.IT] for this version)

Submission history

From: Antoine Gonon [view email]
[v1] Tue, 24 May 2022 07:48:12 GMT (376kb,D)

Link back to: arXiv, form interface, contact.