We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: TimeSHAP: Explaining Recurrent Models through Sequence Perturbations

Abstract: Although recurrent neural networks (RNNs) are state-of-the-art in numerous sequential decision-making tasks, there has been little research on explaining their predictions. In this work, we present TimeSHAP, a model-agnostic recurrent explainer that builds upon KernelSHAP and extends it to the sequential domain. TimeSHAP computes feature-, timestep-, and cell-level attributions. As sequences may be arbitrarily long, we further propose a pruning method that is shown to dramatically decrease both its computational cost and the variance of its attributions. We use TimeSHAP to explain the predictions of a real-world bank account takeover fraud detection RNN model, and draw key insights from its explanations: i) the model identifies important features and events aligned with what fraud analysts consider cues for account takeover; ii) positive predicted sequences can be pruned to only 10% of the original length, as older events have residual attribution values; iii) the most recent input event of positive predictions only contributes on average to 41% of the model's score; iv) notably high attribution to client's age, suggesting a potential discriminatory reasoning, later confirmed as higher false positive rates for older clients.
Comments: Accepted at KDD 2021
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
DOI: 10.1145/3447548.3467166
Cite as: arXiv:2012.00073 [cs.LG]
  (or arXiv:2012.00073v2 [cs.LG] for this version)

Submission history

From: Pedro Saleiro [view email]
[v1] Mon, 30 Nov 2020 19:48:57 GMT (4132kb,D)
[v2] Sat, 26 Jun 2021 09:59:18 GMT (12417kb,D)

Link back to: arXiv, form interface, contact.