English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Pretrained Language Models (LMs) memorize a vast amount of knowledge duringinitial pretraining, including information that may violate the privacy ofpersonal lives and identities. Previous work addressing privacy issues forlanguage models has mostly focused on data preprocessing and differentialprivacy methods, both requiring re-training the underlying LM. We proposeknowledge unlearning as an alternative method to reduce privacy risks for LMspost hoc. We show that simply performing gradient ascent on target tokensequences is effective at forgetting them with little to no degradation ofgeneral language modeling performances for larger LMs; it sometimes evensubstantially improves the underlying LM with just a few iterations. We alsofind that sequential unlearning is better than trying to unlearn all the dataat once and that unlearning is highly dependent on which kind of data (domain)is forgotten. By showing comparisons with a previous data preprocessing methodand a decoding method known to mitigate privacy risks for LMs, we show thatunlearning can give a stronger empirical privacy guarantee in scenarios wherethe data vulnerable to extraction attacks are known a priori while being muchmore efficient and robust. We release the code and dataset needed to replicateour results at https://github.com/joeljang/knowledge-unlearning.

Knowledge Unlearning for Mitigating Privacy Risks in Language Models