English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large pretrained models can be privately fine-tuned to achieve performanceapproaching that of non-private models. A common theme in these results is thesurprising observation that high-dimensional models can achieve favorableprivacy-utility trade-offs. This seemingly contradicts known results on themodel-size dependence of differentially private convex learning and raises thefollowing research question: When does the performance of differentiallyprivate learning not degrade with increasing model size? We identify that themagnitudes of gradients projected onto subspaces is a key factor thatdetermines performance. To precisely characterize this for private convexlearning, we introduce a condition on the objective that we term\emph{restricted Lipschitz continuity} and derive improved bounds for theexcess empirical and population risks that are dimension-independent underadditional conditions. We empirically show that in private fine-tuning of largelanguage models, gradients obtained during fine-tuning are mostly controlled bya few principal components. This behavior is similar to conditions under whichwe obtain dimension-independent bounds in convex settings. Our theoretical andempirical results together provide a possible explanation for recent successesin large-scale private fine-tuning. Code to reproduce our results can be foundat\url{https://github.com/lxuechen/private-transformers/tree/main/examples/classification/spectral_analysis}.

When Does Differentially Private Learning Not Suffer in High Dimensions?