We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: GIST: Distributed Training for Large-Scale Graph Convolutional Networks

Abstract: The graph convolutional network (GCN) is a go-to solution for machine learning on graphs, but its training is notoriously difficult to scale both in terms of graph size and the number of model parameters. Although some work has explored training on large-scale graphs (e.g., GraphSAGE, ClusterGCN, etc.), we pioneer efficient training of large-scale GCN models (i.e., ultra-wide, overparameterized models) with the proposal of a novel, distributed training framework. Our proposed training methodology, called GIST, disjointly partitions the parameters of a GCN model into several, smaller sub-GCNs that are trained independently and in parallel. In addition to being compatible with all GCN architectures and existing sampling techniques for efficient GCN training, GIST i) improves model performance, ii) scales to training on arbitrarily large graphs, iii) decreases wall-clock training time, and iv) enables the training of markedly overparameterized GCN models. Remarkably, with GIST, we train an astonishgly-wide 32,768-dimensional GraphSAGE model, which exceeds the capacity of a single GPU by a factor of 8x, to SOTA performance on the Amazon2M dataset.
Comments: 28 pages, 5 figures, pre-print under review
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC); Optimization and Control (math.OC)
ACM classes: I.2.4
Cite as: arXiv:2102.10424 [cs.LG]
  (or arXiv:2102.10424v4 [cs.LG] for this version)

Submission history

From: Cameron R. Wolfe [view email]
[v1] Sat, 20 Feb 2021 19:25:38 GMT (3105kb,D)
[v2] Wed, 9 Jun 2021 21:11:07 GMT (149kb,D)
[v3] Thu, 1 Jul 2021 13:14:57 GMT (3087kb,D)
[v4] Mon, 14 Mar 2022 14:12:18 GMT (320kb,D)

Link back to: arXiv, form interface, contact.