English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We propose a stochastic variant of the classical Polyak step-size (Polyak,1987) commonly used in the subgradient method. Although computing the Polyakstep-size requires knowledge of the optimal function values, this informationis readily available for typical modern machine learning applications.Consequently, the proposed stochastic Polyak step-size (SPS) is an attractivechoice for setting the learning rate for stochastic gradient descent (SGD). Weprovide theoretical convergence guarantees for SGD equipped with SPS indifferent settings, including strongly convex, convex and non-convex functions.Furthermore, our analysis results in novel convergence guarantees for SGD witha constant step-size. We show that SPS is particularly effective when trainingover-parameterized models capable of interpolating the training data. In thissetting, we prove that SPS enables SGD to converge to the true solution at afast rate without requiring the knowledge of any problem-dependent constants oradditional computational overhead. We experimentally validate our theoreticalresults via extensive experiments on synthetic and real datasets. Wedemonstrate the strong performance of SGD with SPS compared to state-of-the-artoptimization methods when training over-parameterized models.

Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence