References & Citations
Computer Science > Computation and Language
Title: MemBERT: Injecting Unstructured Knowledge into BERT
(Submitted on 2 Sep 2021 (this version), latest version 3 Apr 2024 (v3))
Abstract: Transformers changed modern NLP in many ways. However, they can hardly exploit domain knowledge, and like other blackbox models, they lack interpretability. Unfortunately, structured knowledge injection, in the long run, risks to suffer from a knowledge acquisition bottleneck. We thus propose a memory enhancement of transformer models that makes use of unstructured domain knowledge expressed in plain natural language. An experimental evaluation conducted on two challenging NLP tasks demonstrates that our approach yields better performance and model interpretability than baseline transformer-based architectures.
Submission history
From: Federico Ruggeri [view email][v1] Thu, 2 Sep 2021 09:17:04 GMT (1411kb,D)
[v2] Thu, 12 Jan 2023 13:49:24 GMT (616kb,D)
[v3] Wed, 3 Apr 2024 12:01:46 GMT (513kb,D)
Link back to: arXiv, form interface, contact.