We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CV

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computer Vision and Pattern Recognition

Title: Natural Statistics of Network Activations and Implications for Knowledge Distillation

Abstract: In a matter that is analog to the study of natural image statistics, we study the natural statistics of the deep neural network activations at various layers. As we show, these statistics, similar to image statistics, follow a power law. We also show, both analytically and empirically, that with depth the exponent of this power law increases at a linear rate.
As a direct implication of our discoveries, we present a method for performing Knowledge Distillation (KD). While classical KD methods consider the logits of the teacher network, more recent methods obtain a leap in performance by considering the activation maps. This, however, uses metrics that are suitable for comparing images. We propose to employ two additional loss terms that are based on the spectral properties of the intermediate activation maps. The proposed method obtains state of the art results on multiple image recognition KD benchmarks.
Comments: Accepted to ICIP 2021
Subjects: Computer Vision and Pattern Recognition (cs.CV)
Cite as: arXiv:2106.00368 [cs.CV]
  (or arXiv:2106.00368v1 [cs.CV] for this version)

Submission history

From: Michael Rotman [view email]
[v1] Tue, 1 Jun 2021 10:18:30 GMT (146kb,D)

Link back to: arXiv, form interface, contact.