We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Physics > Data Analysis, Statistics and Probability

Title: Bayesian estimation of information-theoretic metrics for sparsely sampled distributions

Abstract: Estimating the Shannon entropy of a discrete distribution from which we have only observed a small sample is challenging. Estimating other information-theoretic metrics, such as the Kullback-Leibler divergence between two sparsely sampled discrete distributions, is even harder. Existing approaches to address these problems have shortcomings: they are biased, heuristic, work only for some distributions, and/or cannot be applied to all information-theoretic metrics. Here, we propose a fast, semi-analytical estimator for sparsely sampled distributions that is efficient, precise, and general. Its derivation is grounded in probabilistic considerations and uses a hierarchical Bayesian approach to extract as much information as possible from the few observations available. Our approach provides estimates of the Shannon entropy with precision at least comparable to the state of the art, and most often better. It can also be used to obtain accurate estimates of any other information-theoretic metric, including the notoriously challenging Kullback-Leibler divergence. Here, again, our approach performs consistently better than existing estimators.
Comments: 10 pages, 5 figures
Subjects: Data Analysis, Statistics and Probability (physics.data-an); Information Theory (cs.IT); Computation (stat.CO)
Cite as: arXiv:2301.13647 [physics.data-an]
  (or arXiv:2301.13647v2 [physics.data-an] for this version)

Submission history

From: Angelo Piga [view email]
[v1] Tue, 31 Jan 2023 14:07:33 GMT (734kb)
[v2] Wed, 22 Feb 2023 19:25:32 GMT (919kb)

Link back to: arXiv, form interface, contact.