We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: LMSOC: An Approach for Socially Sensitive Pretraining

Abstract: While large-scale pretrained language models have been shown to learn effective linguistic representations for many NLP tasks, there remain many real-world contextual aspects of language that current approaches do not capture. For instance, consider a cloze-test "I enjoyed the ____ game this weekend": the correct answer depends heavily on where the speaker is from, when the utterance occurred, and the speaker's broader social milieu and preferences. Although language depends heavily on the geographical, temporal, and other social contexts of the speaker, these elements have not been incorporated into modern transformer-based language models. We propose a simple but effective approach to incorporate speaker social context into the learned representations of large-scale language models. Our method first learns dense representations of social contexts using graph representation learning algorithms and then primes language model pretraining with these social context representations. We evaluate our approach on geographically-sensitive language-modeling tasks and show a substantial improvement (more than 100% relative lift on MRR) compared to baselines.
Comments: Camera ready version. Accepted to EMNLP 2021 Findings. Code for reproducing the experiments can be found at: this https URL
Subjects: Computation and Language (cs.CL); Computers and Society (cs.CY); Information Retrieval (cs.IR); Machine Learning (cs.LG)
MSC classes: 68T50, 68T07
ACM classes: I.2.7
Cite as: arXiv:2110.10319 [cs.CL]
  (or arXiv:2110.10319v1 [cs.CL] for this version)

Submission history

From: Shubhanshu Mishra [view email]
[v1] Wed, 20 Oct 2021 00:10:37 GMT (5329kb,D)

Link back to: arXiv, form interface, contact.