English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent sharpness-aware minimisation (SAM) is known to find flat minima whichis beneficial for better generalisation with improved robustness. SAMessentially modifies the loss function by reporting the maximum loss valuewithin the small neighborhood around the current iterate. However, it uses theEuclidean ball to define the neighborhood, which can be inaccurate since lossfunctions for neural networks are typically defined over probabilitydistributions (e.g., class predictive probabilities), rendering the parameterspace non Euclidean. In this paper we consider the information geometry of themodel parameter space when defining the neighborhood, namely replacing SAM'sEuclidean balls with ellipsoids induced by the Fisher information. Ourapproach, dubbed Fisher SAM, defines more accurate neighborhood structures thatconform to the intrinsic metric of the underlying statistical manifold. Forinstance, SAM may probe the worst-case loss value at either a too nearby orinappropriately distant point due to the ignorance of the parameter spacegeometry, which is avoided by our Fisher SAM. Another recent Adaptive SAMapproach stretches/shrinks the Euclidean ball in accordance with the scale ofthe parameter magnitudes. This might be dangerous, potentially destroying theneighborhood structure. We demonstrate improved performance of the proposedFisher SAM on several benchmark datasets/tasks.

Fisher SAM: Information Geometry and Sharpness Aware Minimisation