English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Sparsification of neural networks is one of the effective complexityreduction methods to improve efficiency and generalizability. Binarizedactivation offers an additional computational saving for inference. Due tovanishing gradient issue in training networks with binarized activation, coarsegradient (a.k.a. straight through estimator) is adopted in practice. In thispaper, we study the problem of coarse gradient descent (CGD) learning of a onehidden layer convolutional neural network (CNN) with binarized activationfunction and sparse weights. It is known that when the input data is Gaussiandistributed, no-overlap one hidden layer CNN with ReLU activation and generalweight can be learned by GD in polynomial time at high probability inregression problems with ground truth. We propose a relaxed variable splittingmethod integrating thresholding and coarse gradient descent. The sparsity innetwork weight is realized through thresholding during the CGD trainingprocess. We prove that under threshholding of $\ell_1, \ell_0,$ andtransformed-$\ell_1$ penalties, no-overlap binary activation CNN can be learnedwith high probability, and the iterative weights converge to a global limitwhich is a transformation of the true weight under a novel sparsifyingoperation. We found explicit error estimates of sparse weights from the trueweights.

Convergence of a Relaxed Variable Splitting Coarse Gradient Descent Method for Learning Sparse Weight Binarized Activation Neural Networks