English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent breakthroughs in deep neural networks (DNNs) have fueled a tremendousdemand for intelligent edge devices featuring on-site learning, while thepractical realization of such systems remains a challenge due to the limitedresources available at the edge and the required massive training costs forstate-of-the-art (SOTA) DNNs. As reducing precision is one of the mosteffective knobs for boosting training time/energy efficiency, there has been agrowing interest in low-precision DNN training. In this paper, we explore froman orthogonal direction: how to fractionally squeeze out more training costsavings from the most redundant bit level, progressively along the trainingtrajectory and dynamically per input. Specifically, we propose FracTrain thatintegrates (i) progressive fractional quantization which gradually increasesthe precision of activations, weights, and gradients that will not reach theprecision of SOTA static quantized DNN training until the final training stage,and (ii) dynamic fractional quantization which assigns precisions to both theactivations and gradients of each layer in an input-adaptive manner, for only"fractionally" updating layer parameters. Extensive simulations and ablationstudies (six models, four datasets, and three training settings includingstandard, adaptation, and fine-tuning) validate the effectiveness of FracTrainin reducing computational cost and hardware-quantified energy/latency of DNNtraining while achieving a comparable or better (-0.12%~+1.87%) accuracy. Forexample, when training ResNet-74 on CIFAR-10, FracTrain achieves 77.6% and53.5% computational cost and training latency savings, respectively, comparedwith the best SOTA baseline, while achieving a comparable (-0.07%) accuracy.Our codes are available at: https://github.com/RICE-EIC/FracTrain.

FracTrain: Fractionally Squeezing Bit Savings Both Temporally and Spatially for Efficient DNN Training