English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

In the privacy-utility tradeoff of a model trained on benchmark language andvision tasks, remarkable improvements have been widely reported with the use ofpretraining on publicly available data. This is in part due to the benefits oftransfer learning, which is the standard motivation for pretraining innon-private settings. However, the stark contrast in the improvement achievedthrough pretraining under privacy compared to non-private settings suggeststhat there may be a deeper, distinct cause driving these gains. To explain thisphenomenon, we hypothesize that the non-convex loss landscape of a modeltraining necessitates an optimization algorithm to go through two phases. Inthe first, the algorithm needs to select a good "basin" in the loss landscape.In the second, the algorithm solves an easy optimization within that basin. Theformer is a harder problem to solve with private data, while the latter isharder to solve with public data due to a distribution shift or data scarcity.Guided by this intuition, we provide theoretical constructions that provablydemonstrate the separation between private training with and without publicpretraining. Further, systematic experiments on CIFAR10 and LibriSpeech providesupporting evidence for our hypothesis.

Why Is Public Pretraining Necessary for Private Model Training?