English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Deep learning's success has been attributed to the training of large,overparameterized models on massive amounts of data. As this trend continues,model training has become prohibitively costly, requiring access to powerfulcomputing systems to train state-of-the-art networks. A large body of researchhas been devoted to addressing the cost per iteration of training throughvarious model compression techniques like pruning and quantization. Less efforthas been spent targeting the number of iterations. Previous work, such asforget scores and GraNd/EL2N scores, address this problem by identifyingimportant samples within a full dataset and pruning the remaining samples,thereby reducing the iterations per epoch. Though these methods decrease thetraining time, they use expensive static scoring algorithms prior to training.When accounting for the scoring mechanism, the total run time is oftenincreased. In this work, we address this shortcoming with dynamic data pruningalgorithms. Surprisingly, we find that uniform random dynamic pruning canoutperform the prior work at aggressive pruning rates. We attribute this to theexistence of "sometimes" samples -- points that are important to the learneddecision boundary only some of the training time. To better exploit thesubtlety of sometimes samples, we propose two algorithms, based onreinforcement learning techniques, to dynamically prune samples and achieveeven higher accuracy than the random dynamic method. We test all our methodsagainst a full-dataset baseline and the prior work on CIFAR-10 and CIFAR-100,and we can reduce the training time by up to 2x without significant performanceloss. Our results suggest that data pruning should be understood as a dynamicprocess that is closely tied to a model's training trajectory, instead of astatic step based solely on the dataset alone.

Accelerating Deep Learning with Dynamic Data Pruning