English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Efforts to reduce the numerical precision of computations in deep learningtraining have yielded systems that aggressively quantize weights andactivations, yet employ wide high-precision accumulators for partial sums ininner-product operations to preserve the quality of convergence. The absence ofany framework to analyze the precision requirements of partial sumaccumulations results in conservative design choices. This imposes anupper-bound on the reduction of complexity of multiply-accumulate units. Wepresent a statistical approach to analyze the impact of reduced accumulationprecision on deep learning training. Observing that a bad choice foraccumulation precision results in loss of information that manifests itself asa reduction in variance in an ensemble of partial sums, we derive a set ofequations that relate this variance to the length of accumulation and theminimum number of bits needed for accumulation. We apply our analysis to threebenchmark networks: CIFAR-10 ResNet 32, ImageNet ResNet 18 and ImageNetAlexNet. In each case, with accumulation precision set in accordance with ourproposed equations, the networks successfully converge to the single precisionfloating-point baseline. We also show that reducing accumulation precisionfurther degrades the quality of the trained network, proving that our equationsproduce tight bounds. Overall this analysis enables precise tailoring ofcomputation hardware to the application, yielding area- and power-optimalsystems.

Accumulation Bit-Width Scaling For Ultra-Low Precision Training Of Deep Networks