We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Digital Libraries

Title: Web Archive Analytics

Abstract: Web archive analytics is the exploitation of publicly accessible web pages and their evolution for research purposes -- to the extent organizationally possible for researchers. In order to better understand the complexity of this task, the first part of this paper puts the entirety of the world's captured, created, and replicated data (the "Global Datasphere") in relation to other important data sets such as the public internet and its web pages, or what is preserved thereof by the Internet Archive.
Recently, the Webis research group, a network of university chairs to which the authors belong, concluded an agreement with the Internet Archive to download a substantial part of its web archive for research purposes. The second part of the paper in hand describes our infrastructure for processing this data treasure: We will eventually host around 8 PB of web archive data from the Internet Archive and Common Crawl, with the goal of supplementing existing large scale web corpora and forming a non-biased subset of the 30 PB web archive at the Internet Archive.
Comments: 12 pages, 5 figures. Published in the proceedings of INFORMATIK 2020
Subjects: Digital Libraries (cs.DL); Networking and Internet Architecture (cs.NI); Social and Information Networks (cs.SI)
Journal reference: INFORMATIK 2020. Gesellschaft f\"ur Informatik, Bonn. (pp. 61-72)
DOI: 10.18420/inf2020_05
Cite as: arXiv:2107.00893 [cs.DL]
  (or arXiv:2107.00893v1 [cs.DL] for this version)

Submission history

From: Michael Völske [view email]
[v1] Fri, 2 Jul 2021 08:12:50 GMT (1238kb,D)

Link back to: arXiv, form interface, contact.