We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Contextual BERT: Conditioning the Language Model Using a Global State

Abstract: BERT is a popular language model whose main pre-training task is to fill in the blank, i.e., predicting a word that was masked out of a sentence, based on the remaining words. In some applications, however, having an additional context can help the model make the right prediction, e.g., by taking the domain or the time of writing into account. This motivates us to advance the BERT architecture by adding a global state for conditioning on a fixed-sized context. We present our two novel approaches and apply them to an industry use-case, where we complete fashion outfits with missing articles, conditioned on a specific customer. An experimental comparison to other methods from the literature shows that our methods improve personalization significantly.
Comments: Accepted at the TextGraphs-14 workshop at COLING'2020 - The 28th International Conference on Computational Linguistics
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2010.15778 [cs.CL]
  (or arXiv:2010.15778v1 [cs.CL] for this version)

Submission history

From: Timo Denk [view email]
[v1] Thu, 29 Oct 2020 17:25:20 GMT (709kb,D)

Link back to: arXiv, form interface, contact.