References & Citations
Computer Science > Computation and Language
Title: Towards Continual Entity Learning in Language Models for Conversational Agents
(Submitted on 30 Jul 2021 (v1), last revised 15 Sep 2021 (this version, v2))
Abstract: Neural language models (LM) trained on diverse corpora are known to work well on previously seen entities, however, updating these models with dynamically changing entities such as place names, song titles and shopping items requires re-training from scratch and collecting full sentences containing these entities. We aim to address this issue, by introducing entity-aware language models (EALM), where we integrate entity models trained on catalogues of entities into the pre-trained LMs. Our combined language model adaptively adds information from the entity models into the pre-trained LM depending on the sentence context. Our entity models can be updated independently of the pre-trained LM, enabling us to influence the distribution of entities output by the final LM, without any further training of the pre-trained LM. We show significant perplexity improvements on task-oriented dialogue datasets, especially on long-tailed utterances, with an ability to continually adapt to new entities (to an extent).
Submission history
From: Ravi Teja Gadde [view email][v1] Fri, 30 Jul 2021 21:10:09 GMT (274kb,D)
[v2] Wed, 15 Sep 2021 02:43:56 GMT (274kb,D)
Link back to: arXiv, form interface, contact.