We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Reliable Estimation of Kullback-Leibler Divergence by Controlling Discriminator Complexity in the Reproducing Kernel Hilbert Space

Abstract: Several scalable methods to compute the Kullback Leibler (KL) divergence between two distributions using their samples have been proposed and applied in large-scale machine learning models. While they have been found to be unstable, the theoretical root cause of the problem is not clear. In this paper, we study in detail a generative adversarial network based approach that uses a neural network discriminator to estimate KL divergence. We argue that, in such case, high fluctuations in the estimates are a consequence of not controlling the complexity of the discriminator function space. We provide a theoretical underpinning and remedy for this problem through the following contributions. First, we construct a discriminator in the Reproducing Kernel Hilbert Space (RKHS). This enables us to leverage sample complexity and mean embedding to theoretically relate the error probability bound of the KL estimates to the complexity of the neural-net discriminator. Based on this theory, we then present a scalable way to control the complexity of the discriminator for a consistent estimation of KL divergence. We support both our proposed theory and method to control the complexity of the RKHS discriminator in controlled experiments.
Comments: 10 pages, 3 figures
Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Cite as: arXiv:2002.11187 [cs.LG]
  (or arXiv:2002.11187v2 [cs.LG] for this version)

Submission history

From: Sandesh Ghimire [view email]
[v1] Tue, 25 Feb 2020 21:44:52 GMT (667kb,D)
[v2] Fri, 20 Mar 2020 19:06:16 GMT (667kb,D)
[v3] Sat, 26 Sep 2020 22:58:53 GMT (1216kb,D)
[v4] Sat, 4 Sep 2021 22:09:40 GMT (778kb,D)

Link back to: arXiv, form interface, contact.