English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Fine-tuning large pretrained language models on a limited training corpususually suffers from poor generalization. Prior works show that therecently-proposed sharpness-aware minimization (SAM) optimization method canimprove the model generalization. However, SAM adds a perturbation to eachmodel parameter equally (but not all parameters contribute equally to theoptimization of training), which we argue is sub-optimal and will lead toexcessive computation. In this paper, we propose a novel optimizationprocedure, namely FSAM, which introduces a Fisher mask to improve theefficiency and performance of SAM. In short, instead of adding perturbation toall parameters, FSAM uses the Fisher information to identity the importantparameters and formulates a Fisher mask to obtain the sparse perturbation,i.e., making the optimizer focus on these important parameters. Experiments onvarious tasks in GLUE and SuperGLUE benchmarks show that FSAM consistentlyoutperforms the vanilla SAM by 0.67~1.98 average score among four differentpretrained models. We also empirically show that FSAM works well in othercomplex scenarios, e.g., fine-tuning on generation tasks or limited trainingdata. Encouragingly, when training data is limited, FSAM improves the SAM by alarge margin, i.e., up to 15.1.

Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Models