We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Reliable Estimation of KL Divergence using a Discriminator in Reproducing Kernel Hilbert Space

Abstract: Estimating Kullback Leibler (KL) divergence from samples of two distributions is essential in many machine learning problems. Variational methods using neural network discriminator have been proposed to achieve this task in a scalable manner. However, we noted that most of these methods using neural network discriminators suffer from high fluctuations (variance) in estimates and instability in training. In this paper, we look at this issue from statistical learning theory and function space complexity perspective to understand why this happens and how to solve it. We argue that the cause of these pathologies is lack of control over the complexity of the neural network discriminator function and could be mitigated by controlling it. To achieve this objective, we 1) present a novel construction of the discriminator in the Reproducing Kernel Hilbert Space (RKHS), 2) theoretically relate the error probability bound of the KL estimates to the complexity of the discriminator in the RKHS space, 3) present a scalable way to control the complexity (RKHS norm) of the discriminator for a reliable estimation of KL divergence, and 4) prove the consistency of the proposed estimator. In three different applications of KL divergence : estimation of KL, estimation of mutual information and Variational Bayes, we show that by controlling the complexity as developed in the theory, we are able to reduce the variance of KL estimates and stabilize the training
Comments: 27 pages, 3 figures. arXiv admin note: text overlap with arXiv:2002.11187
Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Journal reference: Advances in Neural Information Processing Systems 2021
Cite as: arXiv:2109.14688 [cs.LG]
  (or arXiv:2109.14688v1 [cs.LG] for this version)

Submission history

From: Sandesh Ghimire [view email]
[v1] Wed, 29 Sep 2021 19:50:06 GMT (576kb,D)

Link back to: arXiv, form interface, contact.