English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Recent works on sparse neural network training (sparse training) have shownthat a compelling trade-off between performance and efficiency can be achievedby training intrinsically sparse neural networks from scratch. Existing sparsetraining methods usually strive to find the best sparse subnetwork possible inone single run, without involving any expensive dense or pre-training steps.For instance, dynamic sparse training (DST), is capable of reaching acompetitive performance of dense training by iteratively evolving the sparsetopology during the course of training. In this paper, we argue that it isbetter to allocate the limited resources to create multiple low-loss sparsesubnetworks and superpose them into a stronger one, instead of allocating allresources entirely to find an individual subnetwork. To achieve this, twodesiderata are required: (1) efficiently producing many low-loss subnetworks,the so-called cheap tickets, within one training process limited to thestandard training time used in dense training; (2) effectively superposingthese cheap tickets into one stronger subnetwork. To corroborate ourconjecture, we present a novel sparse training approach, termed Sup-tickets,which can satisfy the above two desiderata concurrently in a singlesparse-to-sparse training process. Across various modern architectures onCIFAR-10/100 and ImageNet, we show that Sup-tickets integrates seamlessly withthe existing sparse training methods and demonstrates consistent performanceimprovement.

Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Training