English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Differentially private stochastic gradient descent (DP-SGD) adds noise togradients in back-propagation, safeguarding training data from privacy leakage,particularly membership inference. It fails to cover (inference-time) threatslike embedding inversion and sensitive attribute inference. It is also costlyin storage and computation when used to fine-tune large pre-trained languagemodels (LMs).  We propose DP-Forward, which directly perturbs embedding matrices in theforward pass of LMs. It satisfies stringent local DP requirements for trainingand inference data. To instantiate it using the smallest matrix-valued noise,we devise an analytic matrix Gaussian~mechanism (aMGM) by drawing possiblynon-i.i.d. noise from a matrix Gaussian distribution. We then investigateperturbing outputs from different hidden (sub-)layers of LMs with aMGM noises.Its utility on three typical tasks almost hits the non-private baseline andoutperforms DP-SGD by up to 7.7pp at a moderate privacy level. It saves3$\times$ time and memory costs compared to DP-SGD with the latest high-speedlibrary. It also reduces the average success rates of embedding inversion andsensitive attribute inference by up to 88pp and 41pp, respectively, whereasDP-SGD fails.

DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass