Current browse context:
cs
Change to browse by:
References & Citations
Statistics > Machine Learning
Title: Variational Representations and Neural Network Estimation of Rényi Divergences
(Submitted on 7 Jul 2020 (v1), last revised 20 Jul 2021 (this version, v4))
Abstract: We derive a new variational formula for the R\'enyi family of divergences, $R_\alpha(Q\|P)$, between probability measures $Q$ and $P$. Our result generalizes the classical Donsker-Varadhan variational formula for the Kullback-Leibler divergence. We further show that this R\'enyi variational formula holds over a range of function spaces; this leads to a formula for the optimizer under very weak assumptions and is also key in our development of a consistency theory for R\'enyi divergence estimators. By applying this theory to neural-network estimators, we show that if a neural network family satisfies one of several strengthened versions of the universal approximation property then the corresponding R\'enyi divergence estimator is consistent. In contrast to density-estimator based methods, our estimators involve only expectations under $Q$ and $P$ and hence are more effective in high dimensional systems. We illustrate this via several numerical examples of neural network estimation in systems of up to 5000 dimensions.
Submission history
From: Jeremiah Birrell [view email][v1] Tue, 7 Jul 2020 22:34:30 GMT (76kb)
[v2] Mon, 5 Oct 2020 15:51:29 GMT (232kb)
[v3] Tue, 27 Apr 2021 13:07:27 GMT (233kb)
[v4] Tue, 20 Jul 2021 16:12:35 GMT (233kb)
Link back to: arXiv, form interface, contact.