Current browse context:
cs.IT
Change to browse by:
References & Citations
Computer Science > Information Theory
Title: Ensemble Estimation of Mutual Information
(Submitted on 27 Jan 2017 (this version), latest version 29 Jul 2021 (v4))
Abstract: We derive the mean squared error convergence rates of kernel density-based plug-in estimators of mutual information measures between two multidimensional random variables $\mathbf{X}$ and $\mathbf{Y}$ for two cases: 1) $\mathbf{X}$ and $\mathbf{Y}$ are both continuous; 2) $\mathbf{X}$ is continuous and $\mathbf{Y}$ is discrete. Using the derived rates, we propose an ensemble estimator of these information measures for the second case by taking a weighted sum of the plug-in estimators with varied bandwidths. The resulting ensemble estimator achieves the $1/N$ parametric convergence rate when the conditional densities of the continuous variables are sufficiently smooth. To the best of our knowledge, this is the first nonparametric mutual information estimator known to achieve the parametric convergence rate for this case, which frequently arises in applications (e.g. variable selection in classification). The estimator is simple to implement as it uses the solution to an offline convex optimization problem and simple plug-in estimators. A central limit theorem is also derived for the ensemble estimator. Ensemble estimators that achieve the parametric rate are also derived for the first case ($\mathbf{X}$ and $\mathbf{Y}$ are both continuous) and another case 3) $\mathbf{X}$ and $\mathbf{Y}$ may have any mixture of discrete and continuous components.
Submission history
From: Kevin Moon [view email][v1] Fri, 27 Jan 2017 15:38:01 GMT (33kb)
[v2] Thu, 5 Sep 2019 22:29:26 GMT (1305kb,D)
[v3] Wed, 30 Jun 2021 05:26:00 GMT (1116kb,D)
[v4] Thu, 29 Jul 2021 17:40:42 GMT (1117kb,D)
Link back to: arXiv, form interface, contact.