English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Quantization of the weights and activations is one of the main methods toreduce the computational footprint of Deep Neural Networks (DNNs) training.Current methods enable 4-bit quantization of the forward phase. However, thisconstitutes only a third of the training process. Reducing the computationalfootprint of the entire training process requires the quantization of theneural gradients, i.e., the loss gradients with respect to the outputs ofintermediate neural layers.  Previous works separately showed that accurate 4-bit quantization of theneural gradients needs to (1) be unbiased and (2) have a log scale. However, noprevious work aimed to combine both ideas, as we do in this work. Specifically,we examine the importance of having unbiased quantization in quantized neuralnetwork training, where to maintain it, and how to combine it with logarithmicquantization. Based on this, we suggest a $\textit{logarithmic unbiasedquantization}$ (LUQ) method to quantize both the forward and backward phases to4-bit, achieving state-of-the-art results in 4-bit training without theoverhead. For example, in ResNet50 on ImageNet, we achieved a degradation of1.1%. We further improve this to a degradation of only 0.32% after three epochsof high precision fine-tuning, combined with a variance reduction method --where both these methods add overhead comparable to previously suggestedmethods.

Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formats