We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

physics.soc-ph

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Physics > Physics and Society

Title: Universal and non-universal text statistics: Clustering coefficient for language identification

Abstract: In this work we analyze statistical properties of 91 relatively small texts in 7 different languages (Spanish, English, French, German, Turkish, Russian, Icelandic) as well as texts with randomly inserted spaces. Despite the size (around 11260 different words), the well known universal statistical laws -- namely Zipf and Herdan-Heap's laws -- are confirmed, and are in close agreement with results obtained elsewhere. We also construct a word co-occurrence network of each text. While the degree distribution is again universal, we note that the distribution of Clustering Coefficients, which depend strongly on the local structure of networks, can be used to differentiate between languages, as well as to distinguish natural languages from random texts.
Comments: 15 pages, 6 figures
Subjects: Physics and Society (physics.soc-ph); Computation and Language (cs.CL)
DOI: 10.1016/j.physa.2019.123905
Cite as: arXiv:1911.08915 [physics.soc-ph]
  (or arXiv:1911.08915v2 [physics.soc-ph] for this version)

Submission history

From: Diego Espitia [view email]
[v1] Mon, 18 Nov 2019 21:39:19 GMT (350kb)
[v2] Sat, 7 Dec 2019 01:26:11 GMT (1961kb)

Link back to: arXiv, form interface, contact.