English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Modern neural networks are undeniably successful. Numerous studies haveinvestigated how the curvature of loss landscapes can affect the quality ofsolutions. In this work we consider the Hessian matrix during network training.We reiterate the connection between the number of "well-determined" or"effective" parameters and the generalisation performance of neural nets, andwe demonstrate its use as a tool for model comparison. By considering the localcurvature, we propose Sharpness Adjusted Number of Effective parameters (SANE),a measure of effective dimensionality for the quality of solutions. We showthat SANE is robust to large learning rates, which represent learning regimesthat are attractive but (in)famously unstable. We provide evidence andcharacterise the Hessian shifts across "loss basins" at large learning rates.Finally, extending our analysis to deeper neural networks, we provide anapproximation to the full-network Hessian, exploiting the natural ordering ofneural weights, and use this approximation to provide extensive empiricalevidence for our claims.

SANE: The phases of gradient descent through Sharpness Adjusted Number of Effective parameters