We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.SE

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Software Engineering

Title: Diet Code is Healthy: Simplifying Programs for Pre-Trained Models of Code

Abstract: Pre-trained code representation models such as CodeBERT have demonstrated superior performance in a variety of software engineering tasks, yet they are often heavy in complexity, quadratically with the length of the input sequence. Our empirical analysis of CodeBERT's attention reveals that CodeBERT pays more attention to certain types of tokens and statements such as keywords and data-relevant statements. Based on these findings, we propose DietCodeBERT, which aims at lightweight leverage of large pre-trained models for source code. DietCodeBERT simplifies the input program of CodeBERT with three strategies, namely, word dropout, frequency filtering, and an attention-based strategy which selects statements and tokens that receive the most attention weights during pre-training. Hence, it gives a substantial reduction in the computational cost without hampering the model performance. Experimental results on two downstream tasks show that DietCodeBERT provides comparable results to CodeBERT with 40% less computational cost in fine-tuning and testing.
Comments: Accepted to be published in ESEC/FSE 2022
Subjects: Software Engineering (cs.SE)
DOI: 10.1145/3540250.3549094
Cite as: arXiv:2206.14390 [cs.SE]
  (or arXiv:2206.14390v2 [cs.SE] for this version)

Submission history

From: Xiaodong Gu [view email]
[v1] Wed, 29 Jun 2022 04:04:38 GMT (742kb,D)
[v2] Wed, 17 Aug 2022 09:23:19 GMT (1463kb,D)
[v3] Tue, 30 Aug 2022 12:14:22 GMT (1463kb,D)
[v4] Tue, 20 Sep 2022 08:18:43 GMT (1498kb,D)
[v5] Mon, 21 Nov 2022 13:31:39 GMT (1498kb,D)

Link back to: arXiv, form interface, contact.