English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Recent research has observed that in machine learning optimization, gradientdescent (GD) often operates at the edge of stability (EoS) [Cohen, et al.,2021], where the stepsizes are set to be large, resulting in non-monotoniclosses induced by the GD iterates. This paper studies the convergence andimplicit bias of constant-stepsize GD for logistic regression on linearlyseparable data in the EoS regime. Despite the presence of local oscillations,we prove that the logistic loss can be minimized by GD with \emph{any} constantstepsize over a long time scale. Furthermore, we prove that with \emph{any}constant stepsize, the GD iterates tend to infinity when projected to amax-margin direction (the hard-margin SVM direction) and converge to a fixedvector that minimizes a strongly convex potential when projected to theorthogonal complement of the max-margin direction. In contrast, we also showthat in the EoS regime, GD iterates may diverge catastrophically under theexponential loss, highlighting the superiority of the logistic loss. Thesetheoretical findings are in line with numerical simulations and complementexisting theories on the convergence and implicit bias of GD for logisticregression, which are only applicable when the stepsizes are sufficientlysmall.

Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability