We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

physics.data-an

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo

Physics > Data Analysis, Statistics and Probability

Title: Learn your entropy from informative data: an axiom ensuring the consistent identification of generalized entropies

Abstract: Shannon entropy, a cornerstone of information theory, statistical physics and inference methods, is uniquely identified by the Shannon-Khinchin or Shore-Johnson axioms. Generalizations of Shannon entropy, motivated by the study of non-extensive or non-ergodic systems, relax some of these axioms and lead to entropy families indexed by certain `entropic' parameters. In general, the selection of these parameters requires pre-knowledge of the system or encounters inconsistencies. Here we introduce a simple axiom for any entropy family: namely, that no entropic parameter can be inferred from a completely uninformative (uniform) probability distribution. When applied to the Uffink-Jizba-Korbel and Hanel-Thurner entropies, the axiom selects only R\'enyi entropy as viable. It also extends consistency with the Maximum Likelihood principle, which can then be generalized to estimate the entropic parameter purely from data, as we confirm numerically. Remarkably, in a generalized maximum-entropy framework the axiom implies that the maximized log-likelihood always equals minus Shannon entropy, even if the inferred probability distribution maximizes a generalized entropy and not Shannon's, solving a series of problems encountered in previous approaches.
Subjects: Data Analysis, Statistics and Probability (physics.data-an); Statistics Theory (math.ST); Methodology (stat.ME)
MSC classes: 62F07, 82B99
ACM classes: J.2; E.4; H.1.1
Cite as: arXiv:2301.05660 [physics.data-an]
  (or arXiv:2301.05660v1 [physics.data-an] for this version)

Submission history

From: Andrea Somazzi [view email]
[v1] Fri, 13 Jan 2023 16:59:35 GMT (56kb)

Link back to: arXiv, form interface, contact.