References & Citations
Computer Science > Computation and Language
Title: Sparse Probability of Agreement
(Submitted on 12 Aug 2022 (v1), last revised 24 Feb 2023 (this version, v2))
Abstract: Measuring inter-annotator agreement is important for annotation tasks, but many metrics require a fully-annotated set of data, where all annotators annotate all samples. We define Sparse Probability of Agreement, SPA, which estimates the probability of agreement when not all annotator-item-pairs are available. We show that under certain conditions, SPA is an unbiased estimator, and we provide multiple weighing schemes for handling data with various degrees of annotation.
Submission history
From: Leon Derczynski [view email][v1] Fri, 12 Aug 2022 08:15:34 GMT (830kb,D)
[v2] Fri, 24 Feb 2023 20:49:54 GMT (842kb,D)
Link back to: arXiv, form interface, contact.