English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Neural network compression has been an increasingly important subject, due toits practical implications in terms of reducing the computational requirementsand its theoretical implications, as there is an explicit connection betweencompressibility and the generalization error. Recent studies have shown thatthe choice of the hyperparameters of stochastic gradient descent (SGD) can havean effect on the compressibility of the learned parameter vector. Even thoughthese results have shed some light on the role of the training dynamics overcompressibility, they relied on unverifiable assumptions and the resultingtheory does not provide a practical guideline due to its implicitness. In thisstudy, we propose a simple modification for SGD, such that the outputs of thealgorithm will be provably compressible without making any nontrivialassumptions. We consider a one-hidden-layer neural network trained with SGD andwe inject additive heavy-tailed noise to the iterates at each iteration. Wethen show that, for any compression rate, there exists a level ofoverparametrization (i.e., the number of hidden units), such that the output ofthe algorithm will be compressible with high probability. To achieve thisresult, we make two main technical contributions: (i) we build on a recentstudy on stochastic analysis and prove a 'propagation of chaos' result withimproved rates for a class of heavy-tailed stochastic differential equations,and (ii) we derive strong-error estimates for their Euler discretization. Wefinally illustrate our approach on experiments, where the results suggest thatthe proposed approach achieves compressibility with a slight compromise fromthe training and test error.

Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD