English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

The local geometry of high dimensional neural network loss landscapes canboth challenge our cherished theoretical intuitions as well as dramaticallyimpact the practical success of neural network training. Indeed recent workshave observed 4 striking local properties of neural loss landscapes onclassification tasks: (1) the landscape exhibits exactly $C$ directions of highpositive curvature, where $C$ is the number of classes; (2) gradient directionsare largely confined to this extremely low dimensional subspace of positiveHessian curvature, leaving the vast majority of directions in weight spaceunexplored; (3) gradient descent transiently explores intermediate regions ofhigher positive curvature before eventually finding flatter minima; (4)training can be successful even when confined to low dimensional {\it random}affine hyperplanes, as long as these hyperplanes intersect a Goldilocks zone ofhigher than average curvature. We develop a simple theoretical model ofgradients and Hessians, justified by numerical experiments on architectures anddatasets used in practice, that {\it simultaneously} accounts for all $4$ ofthese surprising and seemingly unrelated properties. Our unified model providesconceptual insights into the emergence of these properties and makesconnections with diverse topics in neural networks, random matrix theory, andspin glasses, including the neural tangent kernel, BBP phase transitions, andDerrida's random energy model.

Emergent properties of the local geometry of neural loss landscapes