English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Differentially private (DP) optimization is the standard paradigm to learnlarge neural networks that are accurate and privacy-preserving. Thecomputational cost for DP deep learning, however, is notoriously heavy due tothe per-sample gradient clipping. Existing DP implementations are 2-1000X morecostly in time and space complexity than the standard (non-private) training.In this work, we develop a novel Book-Keeping (BK) technique that implementsexisting DP optimizers (thus achieving the same accuracy), with a substantialimprovement on the computational cost. Specifically, BK enables DP training onlarge models and high dimensional data to be roughly as fast and memory-savingas the standard training, whereas previous DP algorithms can be inefficient orincapable of training due to memory error. The computational advantage of BK issupported by the complexity analysis as well as extensive experiments on visionand language tasks. Our implementation achieves state-of-the-art (SOTA)accuracy with very small extra cost: on GPT2 and at almost the same memory cost(<1% overhead), BK has 1.03X the time complexity of the standard training(0.83X training speed in practice), and 0.61X the time complexity of the mostefficient DP implementation (1.36X training speed in practice). We open-sourcethe codebase for the BK algorithm at the FastDP library(https://github.com/awslabs/fast-differential-privacy).

Differentially Private Optimization on Large Model at Small Cost