We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Reliability Testing for Natural Language Processing Systems

Abstract: Questions of fairness, robustness, and transparency are paramount to address before deploying NLP systems. Central to these concerns is the question of reliability: Can NLP systems reliably treat different demographics fairly and function correctly in diverse and noisy environments? To address this, we argue for the need for reliability testing and contextualize it among existing work on improving accountability. We show how adversarial attacks can be reframed for this goal, via a framework for developing reliability tests. We argue that reliability testing -- with an emphasis on interdisciplinary collaboration -- will enable rigorous and targeted testing, and aid in the enactment and enforcement of industry standards.
Comments: Accepted to ACL-IJCNLP 2021 (main conference). Camera-ready version
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY); Neural and Evolutionary Computing (cs.NE)
Cite as: arXiv:2105.02590 [cs.LG]
  (or arXiv:2105.02590v3 [cs.LG] for this version)

Submission history

From: Samson Tan [view email]
[v1] Thu, 6 May 2021 11:24:58 GMT (331kb,D)
[v2] Thu, 13 May 2021 04:17:44 GMT (330kb,D)
[v3] Tue, 1 Jun 2021 03:55:40 GMT (304kb,D)

Link back to: arXiv, form interface, contact.