English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Sparse training is a natural idea to accelerate the training speed of deepneural networks and save the memory usage, especially since large modern neuralnetworks are significantly over-parameterized. However, most of the existingmethods cannot achieve this goal in practice because the chain rule basedgradient (w.r.t. structure parameters) estimators adopted by previous methodsrequire dense computation at least in the backward propagation step. This papersolves this problem by proposing an efficient sparse training method withcompletely sparse forward and backward passes. We first formulate the trainingprocess as a continuous minimization problem under global sparsity constraint.We then separate the optimization process into two steps, corresponding toweight update and structure parameter update. For the former step, we use theconventional chain rule, which can be sparse via exploiting the sparsestructure. For the latter step, instead of using the chain rule based gradientestimators as in existing methods, we propose a variance reduced policygradient estimator, which only requires two forward passes without backwardpropagation, thus achieving completely sparse training. We prove that thevariance of our gradient estimator is bounded. Extensive experimental resultson real-world datasets demonstrate that compared to previous methods, ouralgorithm is much more effective in accelerating the training process, up to anorder of magnitude faster.

Efficient Neural Network Training via Forward and Backward Propagation Sparsification