English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Pre-training has been widely adopted in deep learning to improve modelperformance, especially when the training data for a target task is limited. Inour work, we seek to understand the implications of this training strategy onthe generalization properties of downstream models. More specifically, we askthe following question: how do properties of the pre-training distributionaffect the robustness of a fine-tuned model? The properties we explore includethe label space, label semantics, image diversity, data domains, and dataquantity of the pre-training distribution. We find that the primary factorinfluencing downstream effective robustness (Taori et al., 2020) is dataquantity, while other factors have limited significance. For example, reducingthe number of ImageNet pre-training classes by 4x while increasing the numberof images per class by 4x (that is, keeping total data quantity fixed) does notimpact the robustness of fine-tuned models. We demonstrate our findings onpre-training distributions drawn from various natural and synthetic datasources, primarily using the iWildCam-WILDS distribution shift as a test fordownstream robustness.

On the Connection between Pre-training Data Diversity and Fine-tuning Robustness