English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

A striking observation about iterative magnitude pruning (IMP; Frankle et al.2020) is that $\unicode{x2014}$ after just a few hundred steps of densetraining $\unicode{x2014}$ the method can find a sparse sub-network that can betrained to the same accuracy as the dense network. However, the same does nothold at step 0, i.e. random initialization. In this work, we seek to understandhow this early phase of pre-training leads to a good initialization for IMPboth through the lens of the data distribution and the loss landscape geometry.Empirically we observe that, holding the number of pre-training iterationsconstant, training on a small fraction of (randomly chosen) data suffices toobtain an equally good initialization for IMP. We additionally observe that bypre-training only on "easy" training data, we can decrease the number of stepsnecessary to find a good initialization for IMP compared to training on thefull dataset or a randomly chosen subset. Finally, we identify novel propertiesof the loss landscape of dense networks that are predictive of IMP performance,showing in particular that more examples being linearly mode connected in thedense network correlates well with good initializations for IMP. Combined,these results provide new insight into the role played by the early phasetraining in IMP.

Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks