English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Recently low-bit (e.g., 8-bit) network quantization has been extensivelystudied to accelerate the inference. Besides inference, low-bit training withquantized gradients can further bring more considerable acceleration, since thebackward process is often computation-intensive. Unfortunately, theinappropriate quantization of backward propagation usually makes the trainingunstable and even crash. There lacks a successful unified low-bit trainingframework that can support diverse networks on various tasks. In this paper, wegive an attempt to build a unified 8-bit (INT8) training framework for commonconvolutional neural networks from the aspects of both accuracy and speed.First, we empirically find the four distinctive characteristics of gradients,which provide us insightful clues for gradient quantization. Then, wetheoretically give an in-depth analysis of the convergence bound and derive twoprinciples for stable INT8 training. Finally, we propose two universaltechniques, including Direction Sensitive Gradient Clipping that reduces thedirection deviation of gradients and Deviation Counteractive Learning RateScaling that avoids illegal gradient update along the wrong direction. Theexperiments show that our unified solution promises accurate and efficient INT8training for a variety of networks and tasks, including MobileNetV2,InceptionV3 and object detection that prior studies have never succeeded.Moreover, it enjoys a strong flexibility to run on off-the-shelf hardware, andreduces the training time by 22% on Pascal GPU without too much optimizationeffort. We believe that this pioneering study will help lead the communitytowards a fully unified INT8 training for convolutional neural networks.

Towards Unified INT8 Training for Convolutional Neural Network