We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Information Theory

Title: Universal Densities Exist for Every Finite Reference Measure

Abstract: As it is known, universal codes, which estimate the entropy rate consistently, exist for any stationary ergodic source over a finite alphabet but not over a countably infinite one. We cast the problem of universal coding into the problem of universal densities with respect to a given reference measure on a countably generated measurable space, examples being the counting measure or the Lebesgue measure. We show that universal densities, which estimate the differential entropy rate consistently, exist if the reference measure is finite, which disproves that the assumption of a finite alphabet is necessary in general. To exhibit a universal density, we combine the prediction by partial matching (PPM) code with the non-parametric differential (NPD) entropy rate estimator, putting a prior both over all Markov orders and all quantization levels. The proof of universality applies Barron's asymptotic equipartition for densities and continuity of $f$-divergences for filtrations. As an application, we demonstrate that any universal density induces a strongly consistent Ces\`aro mean estimator of the conditional density given an infinite past, which solves the problem of universal prediction with the $0-1$ loss for a countable alphabet, by the way. We also show that there exists a strongly consistent entropy rate estimator with respect to the Lebesgue measure in the class of stationary ergodic Gaussian processes.
Comments: 21 pages, no figures
Subjects: Information Theory (cs.IT); Statistics Theory (math.ST)
MSC classes: 94A29 (Primary) 62M20 (Secondary)
Cite as: arXiv:2209.11981 [cs.IT]
  (or arXiv:2209.11981v2 [cs.IT] for this version)

Submission history

From: Łukasz Dębowski [view email]
[v1] Sat, 24 Sep 2022 11:29:16 GMT (12kb)
[v2] Sat, 22 Oct 2022 19:48:54 GMT (17kb)

Link back to: arXiv, form interface, contact.