English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent methods have significantly reduced the performance degradation ofBinary Neural Networks (BNNs), but guaranteeing the effective and efficienttraining of BNNs is an unsolved problem. The main reason is that the estimatedgradients produced by the Straight-Through-Estimator (STE) mismatches with thegradients of the real derivatives. In this paper, we provide an explicit convexoptimization example where training the BNNs with the traditionally adaptiveoptimization methods still faces the risk of non-convergence, and identify thatconstraining the range of gradients is critical for optimizing the deep binarymodel to avoid highly suboptimal solutions. For solving above issues, wepropose a BAMSProd algorithm with a key observation that the convergenceproperty of optimizing deep binary model is strongly related to thequantization errors. In brief, it employs an adaptive range constraint via anerrors measurement for smoothing the gradients transition while follows theexponential moving strategy from AMSGrad to avoid errors accumulation duringthe optimization. The experiments verify the corollary of theoreticalconvergence analysis, and further demonstrate that our optimization method canspeed up the convergence about 1:2x and boost the performance of BNNs to asignificant level than the specific binary optimizer about 3:7%, even in ahighly non-convex optimization problem.

BAMSProd: A Step towards Generalizing the Adaptive Optimization Methods to Deep Binary Model