English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large pre-trained transformers have been receiving explosive attention in thepast few years, due to their wide adaptability for numerous downstreamapplications via fine-tuning, but their exponentially increasing parametercounts are becoming a primary hurdle to even just fine-tune them withoutindustry-standard hardware. Recently, Lottery Ticket Hypothesis (LTH) and itsvariants, have been exploited to prune these large pre-trained modelsgenerating subnetworks that can achieve similar performance as their densecounterparts, but LTH pragmatism is enormously inhibited by repetitive fulltraining and pruning routine of iterative magnitude pruning (IMP) which worsenswith increasing model size. Motivated by the recent observations of modelsoups, which suggest that fine-tuned weights of multiple models can be mergedto a better minima, we propose Instant Soup Pruning (ISP) to generate lotteryticket quality subnetworks, using a fraction of the original IMP cost byreplacing the expensive intermediate pruning stages of IMP with computationallyefficient weak mask generation and aggregation routine. More specifically,during the mask generation stage, ISP takes a small handful of iterations usingvarying training protocols and data subsets to generate many weak and noisysubnetworks, and superpose them to average out the noise creating ahigh-quality denoised subnetwork. Our extensive experiments and ablation on twopopular large-scale pre-trained models: CLIP (unexplored in pruning till date)and BERT across multiple benchmark vision and language datasets validate theeffectiveness of ISP compared to several state-of-the-art pruning methods.Codes are available at: \url{https://github.com/VITA-Group/instant_soup}

Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Models