We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Exact Rate-Distortion in Autoencoders via Echo Noise

Abstract: Compression is at the heart of effective representation learning. However, lossy compression is typically achieved through simple parametric models like Gaussian noise to preserve analytic tractability, and the limitations this imposes on learning are largely unexplored. Further, the Gaussian prior assumptions in models such as variational autoencoders (VAEs) provide only an upper bound on the compression rate in general. We introduce a new noise channel, \emph{Echo noise}, that admits a simple, exact expression for mutual information for arbitrary input distributions. The noise is constructed in a data-driven fashion that does not require restrictive distributional assumptions. With its complex encoding mechanism and exact rate regularization, Echo leads to improved bounds on log-likelihood and dominates $\beta$-VAEs across the achievable range of rate-distortion trade-offs. Further, we show that Echo noise can outperform flow-based methods without the need to train additional distributional transformations.
Comments: NeurIPS 2019; updated Gaussian baseline results, added disentanglement
Subjects: Machine Learning (cs.LG); Information Theory (cs.IT); Machine Learning (stat.ML)
Cite as: arXiv:1904.07199 [cs.LG]
  (or arXiv:1904.07199v3 [cs.LG] for this version)

Submission history

From: Rob Brekelmans [view email]
[v1] Mon, 15 Apr 2019 17:22:42 GMT (7946kb,D)
[v2] Sun, 2 Jun 2019 03:21:25 GMT (5364kb,D)
[v3] Thu, 14 Nov 2019 05:41:55 GMT (5749kb,D)

Link back to: arXiv, form interface, contact.