English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Differentially private deep learning has recently witnessed advances incomputational efficiency and privacy-utility trade-off. We explore whetherfurther improvements along the two axes are possible and provide affirmativeanswers leveraging two instantiations of \emph{group-wise clipping}. To reducethe compute time overhead of private learning, we show that \emph{per-layerclipping}, where the gradient of each neural network layer is clippedseparately, allows clipping to be performed in conjunction with backpropagationin differentially private optimization. This results in private learning thatis as memory-efficient and almost as fast per training update as non-privatelearning for many workflows of interest. While per-layer clipping with constantthresholds tends to underperform standard flat clipping, per-layer clippingwith adaptive thresholds matches or outperforms flat clipping under giventraining epoch constraints, hence attaining similar or better task performancewithin less wall time. To explore the limits of scaling (pretrained) models indifferentially private deep learning, we privately fine-tune the 175billion-parameter GPT-3. We bypass scaling challenges associated with clippinggradients that are distributed across multiple devices with \emph{per-deviceclipping} that clips the gradient of each model piece separately on its hostdevice. Privately fine-tuning GPT-3 with per-device clipping achieves a taskperformance at $\epsilon=1$ better than what is attainable by non-privatelyfine-tuning the largest GPT-2 on a summarization task.

Exploring the Limits of Differentially Private Deep Learning with Group-wise Clipping