Current browse context:
stat.ML
Change to browse by:
References & Citations
Statistics > Machine Learning
Title: Information Dropout: learning optimal representations through noise
(Submitted on 4 Nov 2016 (v1), revised 17 Jan 2017 (this version, v2), latest version 12 Feb 2017 (v3))
Abstract: We introduce Information Dropout, a generalization of dropout that is motivated by the Information Bottleneck principle and highlights the way in which injecting noise in the activations can help in learning optimal representations of the data. Information Dropout is rooted in information theoretic principles, it includes as special cases several existing dropout methods, like Gaussian Dropout and Variational Dropout, and, unlike classical dropout, it can learn and build representations that are invariant to nuisances of the data, like occlusions and clutter. When the task is the reconstruction of the input, we show that the information dropout method yields a variational autoencoder as a special case, thus providing a link between representation learning, information theory and variational inference. Our experiments validate the theoretical intuitions behind our method, and we find that information dropout achieves a comparable or better generalization performance than binary dropout, especially on smaller models, since it can automatically adapt the noise to the structure of the network, as well as to the test sample.
Submission history
From: Alessandro Achille [view email][v1] Fri, 4 Nov 2016 12:46:37 GMT (216kb,D)
[v2] Tue, 17 Jan 2017 00:40:19 GMT (256kb,D)
[v3] Sun, 12 Feb 2017 09:26:25 GMT (262kb,D)
Link back to: arXiv, form interface, contact.