We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.DS

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo

Computer Science > Data Structures and Algorithms

Title: Simpler and Better Cardinality Estimators for HyperLogLog and PCSA

Abstract: \emph{Cardinality Estimation} (aka \emph{Distinct Elements}) is a classic problem in sketching with many industrial applications. Although sketching \emph{algorithms} are fairly simple, analyzing the cardinality \emph{estimators} is notoriously difficult, and even today the state-of-the-art sketches such as HyperLogLog and (compressed) \PCSA{} are not covered in graduate level Big Data courses.
In this paper we define a class of \emph{generalized remaining area} (\tGRA) estimators, and observe that HyperLogLog, LogLog, and some estimators for PCSA are merely instantiations of \tGRA{} for various integral values of $\tau$. We then analyze the limiting relative variance of \tGRA{} estimators. It turns out that the standard estimators for HyperLogLog and PCSA can be improved by choosing a \emph{fractional} value of $\tau$. The resulting estimators come \emph{very} close to the Cram\'{e}r-Rao lower bounds for HyperLogLog{} and PCSA derived from their Fisher information. Although the Cram\'{e}r-Rao lower bound \emph{can} be achieved with the Maximum Likelihood Estimator (MLE), the MLE is cumbersome to compute and dynamically update. In contrast, \tGRA{} estimators are trivial to update in constant time.
Our presentation assumes only basic calculus and probability, not any complex analysis~\cite{FlajoletM85,DurandF03,FlajoletFGM07}.
Comments: submitted to sosa23
Subjects: Data Structures and Algorithms (cs.DS)
Cite as: arXiv:2208.10578 [cs.DS]
  (or arXiv:2208.10578v1 [cs.DS] for this version)

Submission history

From: Dingyu Wang [view email]
[v1] Mon, 22 Aug 2022 20:20:43 GMT (166kb,D)

Link back to: arXiv, form interface, contact.