We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:


References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: PECO: Examining Single Sentence Label Leakage in Natural Language Inference Datasets through Progressive Evaluation of Cluster Outliers

Abstract: Building natural language inference (NLI) benchmarks that are both challenging for modern techniques, and free from shortcut biases is difficult. Chief among these biases is "single sentence label leakage," where annotator-introduced spurious correlations yield datasets where the logical relation between (premise, hypothesis) pairs can be accurately predicted from only a single sentence, something that should in principle be impossible. We demonstrate that despite efforts to reduce this leakage, it persists in modern datasets that have been introduced since its 2018 discovery. To enable future amelioration efforts, introduce a novel model-driven technique, the progressive evaluation of cluster outliers (PECO) which enables both the objective measurement of leakage, and the automated detection of subpopulations in the data which maximally exhibit it.
Comments: EACL 2023 14 pages, 8 figures, 5 tables
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2112.09237 [cs.CL]
  (or arXiv:2112.09237v5 [cs.CL] for this version)

Submission history

From: Michael Saxon [view email]
[v1] Thu, 16 Dec 2021 22:49:01 GMT (2086kb,D)
[v2] Wed, 16 Mar 2022 11:49:48 GMT (6788kb,D)
[v3] Wed, 25 May 2022 00:06:26 GMT (6851kb,D)
[v4] Wed, 12 Oct 2022 17:58:36 GMT (6848kb,D)
[v5] Sat, 11 Feb 2023 08:47:56 GMT (6889kb,D)

Link back to: arXiv, form interface, contact.