Multiplication-free neural network training based on adaptive PoT quantization
The current deep neural network training process needs a large number of full-precision multiply-accumulate(MAC)operations,resulting in a situation that the energy consumption of the linear layers(including the convolu-tional layer and the fully connected layer)accounts for the vast majority of the overall energy consumption,reac-hing more than 90%.This work proposes an adaptive layer-wise scaling quantization training method,which can support the replacement of full-precision multiplication in all linear layers with 4-bit fixed-point addition and 1-bit XOR operation.The experimental results show that the above method is superior to the existing methods in terms of energy consumption and accuracy,and can reduce the energy consumption of linear layers by 95.8%in the train-ing process.The convolutional neural networks on ImageNet and the Transformer networks on WMT En-De achieve less than 1%accuracy loss.
neural networkquantizationtraining accelerationlow energy consumption
国家科技期刊平台
NSTL
万方数据
