We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.DC

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Distributed, Parallel, and Cluster Computing

Title: OneFlow: Redesign the Distributed Deep Learning Framework from Scratch

Abstract: Deep learning frameworks such as TensorFlow and PyTorch provide a productive interface for expressing and training a deep neural network (DNN) model on a single device or using data parallelism. Still, they may not be flexible or efficient enough in training emerging large models on distributed devices, which require more sophisticated parallelism beyond data parallelism. Plugins or wrappers have been developed to strengthen these frameworks for model or pipeline parallelism, but they complicate the usage and implementation of distributed deep learning. Aiming at a simple, neat redesign of distributed deep learning frameworks for various parallelism paradigms, we present OneFlow, a novel distributed training framework based on an SBP (split, broadcast and partial-value) abstraction and the actor model. SBP enables much easier programming of data parallelism and model parallelism than existing frameworks, and the actor model provides a succinct runtime mechanism to manage the complex dependencies imposed by resource constraints, data movement and computation in distributed deep learning. We demonstrate the general applicability and efficiency of OneFlow for training various large DNN models with case studies and extensive experiments. The results show that OneFlow outperforms many well-known customized libraries built on top of the state-of-the-art frameworks. The code of OneFlow is available at: this https URL
Subjects: Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as: arXiv:2110.15032 [cs.DC]
  (or arXiv:2110.15032v6 [cs.DC] for this version)

Submission history

From: Jinhui Yuan [view email]
[v1] Thu, 28 Oct 2021 11:32:14 GMT (6212kb)
[v2] Fri, 29 Oct 2021 02:33:23 GMT (6212kb)
[v3] Thu, 27 Jan 2022 06:43:05 GMT (6212kb)
[v4] Tue, 8 Feb 2022 06:46:28 GMT (6212kb)
[v5] Tue, 22 Mar 2022 14:42:54 GMT (6212kb)
[v6] Tue, 19 Apr 2022 11:57:54 GMT (6212kb)

Link back to: arXiv, form interface, contact.