Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Exploring the Potential of Low-bit Training of Convolutional Neural Networks
(Submitted on 4 Jun 2020 (v1), last revised 14 Jul 2021 (this version, v4))
Abstract: In this work, we propose a low-bit training framework for convolutional neural networks, which is built around a novel multi-level scaling (MLS) tensor format. Our framework focuses on reducing the energy consumption of convolution operations by quantizing all the convolution operands to low bit-width format. Specifically, we propose the MLS tensor format, in which the element-wise bit-width can be largely reduced. Then, we describe the dynamic quantization and the low-bit tensor convolution arithmetic to leverage the MLS tensor format efficiently. Experiments show that our framework achieves a superior trade-off between the accuracy and the bit-width than previous low-bit training frameworks. For training a variety of models on CIFAR-10, using 1-bit mantissa and 2-bit exponent is adequate to keep the accuracy loss within $1\%$. And on larger datasets like ImageNet, using 4-bit mantissa and 2-bit exponent is adequate to keep the accuracy loss within $1\%$. Through the energy consumption simulation of the computing units, we can estimate that training a variety of models with our framework could achieve $8.3\sim10.2\times$ and $1.9\sim2.3\times$ higher energy efficiency than training with full-precision and 8-bit floating-point arithmetic, respectively.
Submission history
From: Kai Zhong [view email][v1] Thu, 4 Jun 2020 12:09:35 GMT (387kb,D)
[v2] Tue, 9 Mar 2021 05:59:05 GMT (2294kb,D)
[v3] Thu, 11 Mar 2021 12:10:17 GMT (2294kb,D)
[v4] Wed, 14 Jul 2021 05:54:48 GMT (6876kb,D)
Link back to: arXiv, form interface, contact.