English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Methods for carefully selecting or generating a small set of training data tolearn from, i.e., data pruning, coreset selection, and data distillation, havebeen shown to be effective in reducing the ever-increasing cost of trainingneural networks. Behind this success are rigorously designed strategies foridentifying informative training examples out of large datasets. However, thesestrategies come with additional computational costs associated with subsetselection or data distillation before training begins, and furthermore, manyare shown to even under-perform random sampling in high data compressionregimes. As such, many data pruning, coreset selection, or distillation methodsmay not reduce 'time-to-accuracy', which has become a critical efficiencymeasure of training deep neural networks over large datasets. In this work, werevisit a powerful yet overlooked random sampling strategy to address thesechallenges and introduce an approach called Repeated Sampling of Random Subsets(RSRS or RS2), where we randomly sample the subset of training data for eachepoch of model training. We test RS2 against thirty state-of-the-art datapruning and data distillation methods across four datasets including ImageNet.Our results demonstrate that RS2 significantly reduces time-to-accuracycompared to existing techniques. For example, when training on ImageNet in thehigh-compression regime (using less than 10% of the dataset each epoch), RS2yields accuracy improvements up to 29% compared to competing pruning methodswhile offering a runtime reduction of 7x. Beyond the above meta-study, weprovide a convergence analysis for RS2 and discuss its generalizationcapability. The primary goal of our work is to establish RS2 as a competitivebaseline for future data selection or distillation techniques aimed atefficient training.

Repeated Random Sampling for Minimizing the Time-to-Accuracy of Learning