We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Optimizer Fusion: Efficient Training with Better Locality and Parallelism

Abstract: Machine learning frameworks adopt iterative optimizers to train neural networks. Conventional eager execution separates the updating of trainable parameters from forward and backward computations. However, this approach introduces nontrivial training time overhead due to the lack of data locality and computation parallelism. In this work, we propose to fuse the optimizer with forward or backward computation to better leverage locality and parallelism during training. By reordering the forward computation, gradient calculation, and parameter updating, our proposed method improves the efficiency of iterative optimizers. Experimental results demonstrate that we can achieve an up to 20% training time reduction on various configurations. Since our methods do not alter the optimizer algorithm, they can be used as a general "plug-in" technique to the training process.
Comments: It is published as a paper at the Hardware Aware Efficient Training (HAET) workshop of ICLR 2021. There are 4 pages excluding references and appendices
Subjects: Machine Learning (cs.LG); Mathematical Software (cs.MS)
Cite as: arXiv:2104.00237 [cs.LG]
  (or arXiv:2104.00237v1 [cs.LG] for this version)

Submission history

From: Zixuan Jiang [view email]
[v1] Thu, 1 Apr 2021 03:44:13 GMT (544kb,D)

Link back to: arXiv, form interface, contact.