We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: 92c/MFlops/s, Ultra-Large-Scale Neural-Network Training on a PIII Cluster

Abstract: Artificial neural networks with millions of adjustable parameters and a similar number of training examples are a potential solution for difficult, large-scale pattern recognition problems in areas such as speech and face recognition, classification of large volumes of web data, and finance. The bottleneck is that neural network training involves iterative gradient descent and is extremely computationally intensive. In this paper we present a technique for distributed training of Ultra Large Scale Neural Networks (ULSNN) on Bunyip, a Linux-based cluster of 196 Pentium III processors. To illustrate ULSNN training we describe an experiment in which a neural network with 1.73 million adjustable parameters was trained to recognize machine-printed Japanese characters from a database containing 9 million training patterns. The training runs with a average performance of 163.3 GFlops/s (single precision). With a machine cost of \$150,913, this yields a price/performance ratio of 92.4c/MFlops/s (single precision). For comparison purposes, training using double precision and the ATLAS DGEMM produces a sustained performance of 70 MFlops/s or \$2.16 / MFlop/s (double precision).
Comments: SC '00: Proceedings of the 2000 ACM/IEEE Conference on Supercomputing
Subjects: Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC); Performance (cs.PF); Machine Learning (stat.ML)
Journal reference: ACM/IEEE SC 2000 Conference (SC00)
DOI: 10.1109/SC.2000.10031
Cite as: arXiv:1911.05181 [cs.LG]
  (or arXiv:1911.05181v1 [cs.LG] for this version)

Submission history

From: Jonathan Baxter [view email]
[v1] Tue, 12 Nov 2019 22:57:09 GMT (290kb,D)

Link back to: arXiv, form interface, contact.