We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Analysis of Discriminator in RKHS Function Space for Kullback-Leibler Divergence Estimation

Abstract: Several scalable sample-based methods to compute the Kullback Leibler (KL) divergence between two distributions have been proposed and applied in large-scale machine learning models. While they have been found to be unstable, the theoretical root cause of the problem is not clear. In this paper, we study a generative adversarial network based approach that uses a neural network discriminator to estimate KL divergence. We argue that, in such case, high fluctuations in the estimates are a consequence of not controlling the complexity of the discriminator function space. We provide a theoretical underpinning and remedy for this problem by first constructing a discriminator in the Reproducing Kernel Hilbert Space (RKHS). This enables us to leverage sample complexity and mean embedding to theoretically relate the error probability bound of the KL estimates to the complexity of the discriminator in RKHS. Based on this theory, we then present a scalable way to control the complexity of the discriminator for a reliable estimation of KL divergence. We support both our proposed theory and method to control the complexity of the RKHS discriminator through controlled experiments.
Comments: 15 pages, 3 figures
Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Cite as: arXiv:2002.11187 [cs.LG]
  (or arXiv:2002.11187v4 [cs.LG] for this version)

Submission history

From: Sandesh Ghimire [view email]
[v1] Tue, 25 Feb 2020 21:44:52 GMT (667kb,D)
[v2] Fri, 20 Mar 2020 19:06:16 GMT (667kb,D)
[v3] Sat, 26 Sep 2020 22:58:53 GMT (1216kb,D)
[v4] Sat, 4 Sep 2021 22:09:40 GMT (778kb,D)

Link back to: arXiv, form interface, contact.