English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent studies of gradient descent with large step sizes have shown thatthere is often a regime with an initial increase in the largest eigenvalue ofthe loss Hessian (progressive sharpening), followed by a stabilization of theeigenvalue near the maximum value which allows convergence (edge of stability).These phenomena are intrinsically non-linear and do not happen for models inthe constant Neural Tangent Kernel (NTK) regime, for which the predictivefunction is approximately linear in the parameters. As such, we consider thenext simplest class of predictive models, namely those that are quadratic inthe parameters, which we call second-order regression models. For quadraticobjectives in two dimensions, we prove that this second-order regression modelexhibits progressive sharpening of the NTK eigenvalue towards a value thatdiffers slightly from the edge of stability, which we explicitly compute. Inhigher dimensions, the model generically shows similar behavior, even withoutthe specific structure of a neural network, suggesting that progressivesharpening and edge-of-stability behavior aren't unique features of neuralnetworks, and could be a more general property of discrete learning algorithmsin high-dimensional non-linear models.

Second-order regression models exhibit progressive sharpening to the edge of stability