We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Gender Biases Unexpectedly Fluctuate in the Pre-training Stage of Masked Language Models

Abstract: Masked language models pick up gender biases during pre-training. Such biases are usually attributed to a certain model architecture and its pre-training corpora, with the implicit assumption that other variations in the pre-training process, such as the choices of the random seed or the stopping point, have no effect on the biases measured. However, we show that severe fluctuations exist at the fundamental level of individual templates, invalidating the assumption. Further against the intuition of how humans acquire biases, these fluctuations are not correlated with the certainty of the predicted pronouns or the profession frequencies in pre-training corpora. We release our code and data to benefit future research.
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2211.14639 [cs.CL]
  (or arXiv:2211.14639v1 [cs.CL] for this version)

Submission history

From: Kenan Tang [view email]
[v1] Sat, 26 Nov 2022 18:43:05 GMT (8735kb,D)

Link back to: arXiv, form interface, contact.