We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Be More Active! Understanding the Differences between Mean and Sampled Representations of Variational Autoencoders

Abstract: The ability of Variational Autoencoders to learn disentangled representations has made them appealing for practical applications. However, their mean representations, which are generally used for downstream tasks, have recently been shown to be more correlated than their sampled counterpart, on which disentanglement is usually measured. In this paper, we refine this observation through the lens of selective posterior collapse, which states that only a subset of the learned representations, the active variables, is encoding useful information while the rest (the passive variables) is discarded. We first extend the existing definition, originally proposed for sampled representations, to mean representations and show that active variables are equally disentangled in both representations. Based on this new definition and the pre-trained models from disentanglement lib, we then isolate the passive variables and show that they are responsible for the discrepancies between mean and sampled representations. Specifically, passive variables exhibit high correlation scores with other variables in mean representations while being fully uncorrelated in sampled ones. We thus conclude that despite what their higher correlation might suggest, mean representations are still good candidates for downstream tasks applications. However, it may be beneficial to remove their passive variables, especially when used with models sensitive to correlated features.
Comments: the main paper of 26 pages plus an appendix; 97 pages in total
Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
ACM classes: I.2.6; G.3
Cite as: arXiv:2109.12679 [cs.LG]
  (or arXiv:2109.12679v2 [cs.LG] for this version)

Submission history

From: Lisa Bonheme [view email]
[v1] Sun, 26 Sep 2021 19:04:57 GMT (1490kb,D)
[v2] Wed, 29 Sep 2021 09:08:45 GMT (1488kb,D)

Link back to: arXiv, form interface, contact.