Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: A Survey on Efficient Training of Transformers
(Submitted on 2 Feb 2023 (v1), last revised 4 May 2023 (this version, v3))
Abstract: Recent advances in Transformers have come with a huge requirement on computing resources, highlighting the importance of developing efficient training techniques to make Transformer training faster, at lower cost, and to higher accuracy by the efficient use of computation and memory resources. This survey provides the first systematic overview of the efficient training of Transformers, covering the recent progress in acceleration arithmetic and hardware, with a focus on the former. We analyze and compare methods that save computation and memory costs for intermediate tensors during training, together with techniques on hardware/algorithm co-design. We finally discuss challenges and promising areas for future research.
Submission history
From: Bohan Zhuang [view email][v1] Thu, 2 Feb 2023 13:58:18 GMT (61kb,D)
[v2] Wed, 22 Feb 2023 13:06:27 GMT (75kb,D)
[v3] Thu, 4 May 2023 01:23:12 GMT (76kb,D)
Link back to: arXiv, form interface, contact.