English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Understanding the optimization dynamics of neural networks is necessary forclosing the gap between theory and practice. Stochastic first-orderoptimization algorithms are known to efficiently locate favorable minima indeep neural networks. This efficiency, however, contrasts with the non-convexand seemingly complex structure of neural loss landscapes. In this study, wedelve into the fundamental geometric properties of sampled gradients alongoptimization paths. We focus on two key quantities, which appear in therestricted secant inequality and error bound. Both hold high significance forfirst-order optimization. Our analysis reveals that these quantities exhibitpredictable, consistent behavior throughout training, despite the stochasticityinduced by sampling minibatches. Our findings suggest that not only dooptimization trajectories never encounter significant obstacles, but they alsomaintain stable dynamics during the majority of training. These observedproperties are sufficiently expressive to theoretically guarantee linearconvergence and prescribe learning rate schedules mirroring empiricalpractices. We conduct our experiments on image classification, semanticsegmentation and language modeling across different batch sizes, networkarchitectures, datasets, optimizers, and initialization seeds. We discuss theimpact of each factor. Our work provides novel insights into the properties ofneural network loss functions, and opens the door to theoretical frameworksmore relevant to prevalent practice.

No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths