English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Utilizing massive web-scale datasets has led to unprecedented performancegains in machine learning models, but also imposes outlandish computerequirements for their training. In order to improve training and dataefficiency, we here push the limits of pruning large-scale multimodal datasetsfor training CLIP-style models. Today's most effective pruning method onImageNet clusters data samples into separate concepts according to theirembedding and prunes away the most prototypical samples. We scale this approachto LAION and improve it by noting that the pruning rate should beconcept-specific and adapted to the complexity of the concept. Using a simpleand intuitive complexity measure, we are able to reduce the training cost to aquarter of regular training. By filtering from the LAION dataset, we find thattraining on a smaller set of high-quality data can lead to higher performancewith significantly lower training costs. More specifically, we are able tooutperform the LAION-trained OpenCLIP-ViT-B32 model on ImageNet zero-shotaccuracy by 1.1p.p. while only using 27.7% of the data and training compute.Despite a strong reduction in training cost, we also see improvements onImageNet dist. shifts, retrieval tasks and VTAB. On the DataComp Mediumbenchmark, we achieve a new state-of-the-art ImageNet zero-shot accuracy and acompetitive average zero-shot accuracy on 38 evaluation tasks.

Effective pruning of web-scale datasets based on complexity of concept clusters