References & Citations
Computer Science > Distributed, Parallel, and Cluster Computing
Title: Clairvoyant Prefetching for Distributed Machine Learning I/O
(Submitted on 21 Jan 2021 (v1), last revised 10 Jun 2021 (this version, v2))
Abstract: I/O is emerging as a major bottleneck for machine learning training, especially in distributed environments. Indeed, at large scale, I/O takes as much as 85% of training time. Addressing this I/O bottleneck necessitates careful optimization, as optimal data ingestion pipelines differ between systems, and require a delicate balance between access to local storage, external filesystems, and remote nodes. We introduce NoPFS, a machine learning I/O middleware, which provides a scalable, flexible, and easy-to-use solution to the I/O bottleneck. NoPFS uses clairvoyance: Given the seed generating the random access pattern for training with SGD, it can exactly predict when and where a sample will be accessed. We combine this with an analysis of access patterns and a performance model to provide distributed caching policies that adapt to different datasets and storage hierarchies. NoPFS reduces I/O times and improves end-to-end training by up to 5.4x on the ImageNet-1k, ImageNet-22k, and CosmoFlow datasets.
Submission history
From: Nikoli Dryden [view email][v1] Thu, 21 Jan 2021 17:21:42 GMT (675kb,D)
[v2] Thu, 10 Jun 2021 12:28:20 GMT (645kb,D)
Link back to: arXiv, form interface, contact.