We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo

Computer Science > Computation and Language

Title: Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalization

Abstract: Prompt tuning is a parameter-efficient method, which learns soft prompts and conditions frozen language models to perform specific downstream tasks. Though effective, prompt tuning under few-shot settings on the one hand heavily relies on a good initialization of soft prompts. On the other hand, it can easily result in overfitting. Existing works leverage pre-training or supervised meta-learning to initialize soft prompts but they cannot data-efficiently generalize to unseen downstream tasks. To address the above problems, this paper proposes a novel Self-sUpervised meta-Prompt learning framework with meta-gradient Regularization for few-shot generalization (SUPMER). We first design a set of self-supervised anchor meta-training tasks with different task formats and further enrich the task distribution with curriculum-based task augmentation. Then a novel meta-gradient regularization method is integrated into meta-prompt learning. It meta-learns to transform the raw gradients during few-shot learning into a domain-generalizable direction, thus alleviating the problem of overfitting. Extensive experiments show that SUPMER achieves better performance for different few-shot downstream tasks, and also exhibits a stronger domain generalization ability.
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as: arXiv:2303.12314 [cs.CL]
  (or arXiv:2303.12314v1 [cs.CL] for this version)

Submission history

From: Kaihang Pan [view email]
[v1] Wed, 22 Mar 2023 05:04:21 GMT (7022kb,D)
[v2] Tue, 28 Mar 2023 13:56:07 GMT (7021kb,D)
[v3] Sun, 21 May 2023 07:18:54 GMT (7021kb,D)
[v4] Mon, 23 Oct 2023 12:43:35 GMT (268kb,D)

Link back to: arXiv, form interface, contact.