We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Towards Automated Error Analysis: Learning to Characterize Errors

Abstract: Characterizing the patterns of errors that a system makes helps researchers focus future development on increasing its accuracy and robustness. We propose a novel form of "meta learning" that automatically learns interpretable rules that characterize the types of errors that a system makes, and demonstrate these rules' ability to help understand and improve two NLP systems. Our approach works by collecting error cases on validation data, extracting meta-features describing these samples, and finally learning rules that characterize errors using these features. We apply our approach to VilBERT, for Visual Question Answering, and RoBERTa, for Common Sense Question Answering. Our system learns interpretable rules that provide insights into systemic errors these systems make on the given tasks. Using these insights, we are also able to "close the loop" and modestly improve performance of these systems.
Comments: 12 pages, 11 figures
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as: arXiv:2201.05017 [cs.CL]
  (or arXiv:2201.05017v3 [cs.CL] for this version)

Submission history

From: Tong Gao [view email]
[v1] Thu, 13 Jan 2022 15:08:44 GMT (17450kb,D)
[v2] Fri, 14 Jan 2022 01:44:28 GMT (17450kb,D)
[v3] Mon, 14 Feb 2022 01:39:08 GMT (17450kb,D)

Link back to: arXiv, form interface, contact.