English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

When applied to large-scale learning problems, the conventional wisdom onprivacy-preserving deep learning, known as Differential Private StochasticGradient Descent (DP-SGD), has met with limited success due to significantperformance degradation and high memory overhead when compared to thenon-privacy counterpart. We show how to mitigate the performance drop byreplacing the DP-SGD with a novel DP Forward-Propagation (DP-FP) followed by anoff-the-shelf non-DP optimizer. Our DP-FP employs novel (1) representationclipping followed by noise addition in the forward propagation stage, as wellas (2) micro-batch construction via subsampling to achieve DP amplification andreduce noise power to $1/M$, where $M$ is the number of micro-batch in a step.When training a classification model, our DP-FP with all of theprivacy-preserving operations on the representation is innately free ofgradient bias, total noise proportionally to model size, and memory issues inDP-SGD. As a result, our DP-FP outperforms cutting-edge DP-SGD while retainingthe same level of privacy, and it approaches non-private baselines andsignificantly outperforms state-of-the-art DP-SGD variants. When applied toRoBERTa-large on four downstream tasks, for example, DP-FP achieves an averageaccuracy of 91.34\% with privacy budgets less than 3, representing a 3.81\%performance improvement over the state-of-the-art DP-SGD and only a 0.9\% losscompared to the non-private baseline but with a significantly lower privacyleakage risk.

DP-FP: Differentially Private Forward Propagation for Large Models