We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalization

Abstract: Fine-tuning large pre-trained models with task-specific data has achieved great success in NLP. However, it has been demonstrated that the majority of information within the self-attention networks is redundant and not utilized effectively during the fine-tuning stage. This leads to inferior results when generalizing the obtained models to out-of-domain distributions. To this end, we propose a simple yet effective data augmentation technique, HiddenCut, to better regularize the model and encourage it to learn more generalizable features. Specifically, contiguous spans within the hidden space are dynamically and strategically dropped during training. Experiments show that our HiddenCut method outperforms the state-of-the-art augmentation methods on the GLUE benchmark, and consistently exhibits superior generalization performances on out-of-distribution and challenging counterexamples. We have publicly released our code at this https URL
Comments: ACL 2021
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2106.00149 [cs.CL]
  (or arXiv:2106.00149v1 [cs.CL] for this version)

Submission history

From: Jiaao Chen [view email]
[v1] Mon, 31 May 2021 23:57:43 GMT (742kb,D)

Link back to: arXiv, form interface, contact.