References & Citations
Computer Science > Computation and Language
Title: To what extent do human explanations of model behavior align with actual model behavior?
(Submitted on 24 Dec 2020 (v1), last revised 16 Sep 2021 (this version, v2))
Abstract: Given the increasingly prominent role NLP models (will) play in our lives, it is important for human expectations of model behavior to align with actual model behavior. Using Natural Language Inference (NLI) as a case study, we investigate the extent to which human-generated explanations of models' inference decisions align with how models actually make these decisions. More specifically, we define three alignment metrics that quantify how well natural language explanations align with model sensitivity to input words, as measured by integrated gradients. Then, we evaluate eight different models (the base and large versions of BERT, RoBERTa and ELECTRA, as well as anRNN and bag-of-words model), and find that the BERT-base model has the highest alignment with human-generated explanations, for all alignment metrics. Focusing in on transformers, we find that the base versions tend to have higher alignment with human-generated explanations than their larger counterparts, suggesting that increasing the number of model parameters leads, in some cases, to worse alignment with human explanations. Finally, we find that a model's alignment with human explanations is not predicted by the model's accuracy, suggesting that accuracy and alignment are complementary ways to evaluate models.
Submission history
From: Grusha Prasad [view email][v1] Thu, 24 Dec 2020 17:40:06 GMT (6190kb,D)
[v2] Thu, 16 Sep 2021 20:54:26 GMT (5491kb,D)
Link back to: arXiv, form interface, contact.