English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently, flat minima are proven to be effective for improving generalizationand sharpness-aware minimization (SAM) achieves state-of-the-art performance.Yet the current definition of flatness discussed in SAM and its follow-ups arelimited to the zeroth-order flatness (i.e., the worst-case loss within aperturbation radius). We show that the zeroth-order flatness can beinsufficient to discriminate minima with low generalization error from thosewith high generalization error both when there is a single minimum or multipleminima within the given perturbation radius. Thus we present first-orderflatness, a stronger measure of flatness focusing on the maximal gradient normwithin a perturbation radius which bounds both the maximal eigenvalue ofHessian at local minima and the regularization function of SAM. We also presenta novel training procedure named Gradient norm Aware Minimization (GAM) to seekminima with uniformly small curvature across all directions. Experimentalresults show that GAM improves the generalization of models trained withcurrent optimizers such as SGD and AdamW on various datasets and networks.Furthermore, we show that GAM can help SAM find flatter minima and achievebetter generalization.

Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization