English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Differentially private stochastic gradient descent (DP-SGD) is known to havepoorer training and test performance on large neural networks, compared toordinary stochastic gradient descent (SGD). In this paper, we perform adetailed study and comparison of the two processes and unveil several newinsights. By comparing the behavior of the two processes separately in earlyand late epochs, we find that while DP-SGD makes slower progress in earlystages, it is the behavior in the later stages that determines the end result.This separate analysis of the clipping and noise addition steps of DP-SGD showsthat while noise introduces errors to the process, gradient descent can recoverfrom these errors when it is not clipped, and clipping appears to have a largerimpact than noise. These effects are amplified in higher dimensions (largeneural networks), where the loss basin occupies a lower dimensional space. Weargue theoretically and using extensive experiments that magnitude pruning canbe a suitable dimension reduction technique in this regard, and find that heavypruning can improve the test accuracy of DPSGD.

Inference and Interference: The Role of Clipping, Pruning and Loss Landscapes in Differentially Private Stochastic Gradient Descent