English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Here we develop variants of SGD (stochastic gradient descent) with anadaptive step size that make use of the sampled loss values. In particular, wefocus on solving a finite sum-of-terms problem, also known as empirical riskminimization. We first detail an idealized adaptive method called$\texttt{SPS}_+$ that makes use of the sampled loss values and assumesknowledge of the sampled loss at optimality. This $\texttt{SPS}_+$ is a minormodification of the SPS (Stochastic Polyak Stepsize) method, where the stepsize is enforced to be positive. We then show that $\texttt{SPS}_+$ achievesthe best known rates of convergence for SGD in the Lipschitz non-smooth. Wethen move onto to develop $\texttt{FUVAL}$, a variant of $\texttt{SPS}_+$ wherethe loss values at optimality are gradually learned, as opposed to being given.We give three viewpoints of $\texttt{FUVAL}$, as a projection based method, asa variant of the prox-linear method, and then as a particular online SGDmethod. We then present a convergence analysis of $\texttt{FUVAL}$ andexperimental results. The shortcomings of our work is that the convergenceanalysis of $\texttt{FUVAL}$ shows no advantage over SGD. Another shortcommingis that currently only the full batch version of $\texttt{FUVAL}$ shows a minoradvantages of GD (Gradient Descent) in terms of sensitivity to the step size.The stochastic version shows no clear advantage over SGD. We conjecture thatlarge mini-batches are required to make $\texttt{FUVAL}$ competitive.  Currently the new $\texttt{FUVAL}$ method studied in this paper does notoffer any clear theoretical or practical advantage. We have chosen to make thisdraft available online nonetheless because of some of the analysis techniqueswe use, such as the non-smooth analysis of $\texttt{SPS}_+$, and also to showan apparently interesting approach that currently does not work.

Function Value Learning: Adaptive Learning Rates Based on the Polyak Stepsize and Function Splitting in ERM