English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

This paper proposes an algorithm (RMDA) for training neural networks (NNs)with a regularization term for promoting desired structures. RMDA does notincur computation additional to proximal SGD with momentum, and achievesvariance reduction without requiring the objective function to be of thefinite-sum form. Through the tool of manifold identification from nonlinearoptimization, we prove that after a finite number of iterations, all iteratesof RMDA possess a desired structure identical to that induced by theregularizer at the stationary point of asymptotic convergence, even in thepresence of engineering tricks like data augmentation and dropout thatcomplicate the training process. Experiments on training NNs with structuredsparsity confirm that variance reduction is necessary for such anidentification, and show that RMDA thus significantly outperforms existingmethods for this task. For unstructured sparsity, RMDA also outperforms astate-of-the-art pruning method, validating the benefits of training structuredNNs through regularization.

Training Structured Neural Networks Through Manifold Identification and Variance Reduction