Open AccessQuantization of Weights of Neural Networks with Negligible Decreasing of Prediction Accuracy
Author(s) -
Zoran Perić,
Bojan Denić,
Milan Savić,
Milan R. Dinčić,
Darko Mihajlov
Publication year - 2021
Publication title -
informacinės technologijos ir valdymas
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.286
H-Index - 19
eISSN - 2335-884X
pISSN - 1392-124X
DOI - 10.5755/j01.itc.50.3.28468
Subject(s) - quantization (signal processing) , algorithm , artificial neural network , mathematics , mean squared error , computer science , variance (accounting) , sample size determination , statistics , artificial intelligence , accounting , business
Quantization and compression of neural network parameters using the uniform scalar quantization is carried out in this paper. The attractiveness of the uniform scalar quantizer is reflected in a low complexity and relatively good performance, making it the most popular quantization model. We present a design approach for the memoryless Laplacian source with zero-mean and unit variance, which is based on iterative rule and uses the minimal mean-squared error distortion as a performance criterion. In addition, we derive closed-form expressions for SQNR (Signal to Quantization Noise Ratio) in a wide dynamic range of variance of input data. To show effectiveness on real data, the proposed quantizer is used to compress the weights of neural networks using bit rates from 9 to 16 bps (bits/sample) instead of standardly used 32 bps full precision bit rate. The impact of weights compression on the NN (neural network) performance is analyzed, indicating good matching with the theoretical results and showing negligible decreasing of the prediction accuracy of the NN even in the case of high variance-mismatch between the variance of NN weights and the variance used for the design of quantizer, if the value of the bit-rate is properly chosen according to the rule proposed in the paper.