English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Binary neural networks (BNNs) have received increasing attention due to theirsuperior reductions of computation and memory. Most existing works focus oneither lessening the quantization error by minimizing the gap between thefull-precision weights and their binarization or designing a gradientapproximation to mitigate the gradient mismatch, while leaving the "deadweights" untouched. This leads to slow convergence when training BNNs. In thispaper, for the first time, we explore the influence of "dead weights" whichrefer to a group of weights that are barely updated during the training ofBNNs, and then introduce rectified clamp unit (ReCU) to revive the "deadweights" for updating. We prove that reviving the "dead weights" by ReCU canresult in a smaller quantization error. Besides, we also take into account theinformation entropy of the weights, and then mathematically analyze why theweight standardization can benefit BNNs. We demonstrate the inherentcontradiction between minimizing the quantization error and maximizing theinformation entropy, and then propose an adaptive exponential scheduler toidentify the range of the "dead weights". By considering the "dead weights",our method offers not only faster BNN training, but also state-of-the-artperformance on CIFAR-10 and ImageNet, compared with recent methods. Code can beavailable at https://github.com/z-hXu/ReCU.

ReCU: Reviving the Dead Weights in Binary Neural Networks