English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The success of lottery ticket initializations (Frankle and Carbin, 2019)suggests that small, sparsified networks can be trained so long as the networkis initialized appropriately. Unfortunately, finding these "winning ticket"initializations is computationally expensive. One potential solution is toreuse the same winning tickets across a variety of datasets and optimizers.However, the generality of winning ticket initializations remains unclear.Here, we attempt to answer this question by generating winning tickets for onetraining configuration (optimizer and dataset) and evaluating their performanceon another configuration. Perhaps surprisingly, we found that, within thenatural images domain, winning ticket initializations generalized across avariety of datasets, including Fashion MNIST, SVHN, CIFAR-10/100, ImageNet, andPlaces365, often achieving performance close to that of winning ticketsgenerated on the same dataset. Moreover, winning tickets generated using largerdatasets consistently transferred better than those generated using smallerdatasets. We also found that winning ticket initializations generalize acrossoptimizers with high performance. These results suggest that winning ticketinitializations generated by sufficiently large datasets contain inductivebiases generic to neural networks more broadly which improve training acrossmany settings and provide hope for the development of better initializationmethods.

One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers