References & Citations
Computer Science > Computation and Language
Title: Memorisation versus Generalisation in Pre-trained Language Models
(Submitted on 16 Apr 2021 (v1), last revised 15 Mar 2022 (this version, v2))
Abstract: State-of-the-art pre-trained language models have been shown to memorise facts and perform well with limited amounts of training data. To gain a better understanding of how these models learn, we study their generalisation and memorisation capabilities in noisy and low-resource scenarios. We find that the training of these models is almost unaffected by label noise and that it is possible to reach near-optimal results even on extremely noisy datasets. However, our experiments also show that they mainly learn from high-frequency patterns and largely fail when tested on low-resource tasks such as few-shot learning and rare entity recognition. To mitigate such limitations, we propose an extension based on prototypical networks that improves performance in low-resource named entity recognition tasks.
Submission history
From: Michael Tänzer Mr [view email][v1] Fri, 16 Apr 2021 18:53:19 GMT (2215kb,D)
[v2] Tue, 15 Mar 2022 01:14:16 GMT (2313kb,D)
Link back to: arXiv, form interface, contact.