We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.AP

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Applications

Title: Bias in Zipf's Law Estimators

Abstract: The prevailing maximum likelihood estimators for inferring power law models from rank-frequency data are biased. The source of this bias is an inappropriate likelihood function. The correct likelihood function is derived and shown to be computationally intractable. A more computationally efficient method of approximate Bayesian computation (ABC) is explored. This method is shown to have less bias for data generated from idealised rank-frequency Zipfian distributions. However, the existing estimators and the ABC estimator described here assume that words are drawn from a simple probability distribution, while language is a much more complex process. We show that this false assumption leads to continued biases when applying any of these methods to natural language to estimate Zipf exponents. We recommend that researchers be aware of these biases when investigating power laws in rank-frequency data.
Comments: 15 pages, 11 figures
Subjects: Applications (stat.AP)
Cite as: arXiv:2008.00903 [stat.AP]
  (or arXiv:2008.00903v4 [stat.AP] for this version)

Submission history

From: Charlie Pilgrim [view email]
[v1] Mon, 3 Aug 2020 14:33:21 GMT (608kb,D)
[v2] Tue, 26 Jan 2021 16:18:37 GMT (1690kb,D)
[v3] Wed, 12 May 2021 15:17:01 GMT (1693kb,D)
[v4] Mon, 26 Jul 2021 10:50:39 GMT (1694kb,D)

Link back to: arXiv, form interface, contact.