English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Stochastic Gradient Descent (SGD) is a central tool in machine learning. Weprove that SGD converges to zero loss, even with a fixed (non-vanishing)learning rate - in the special case of homogeneous linear classifiers withsmooth monotone loss functions, optimized on linearly separable data. Previousworks assumed either a vanishing learning rate, iterate averaging, or lossassumptions that do not hold for monotone loss functions used forclassification, such as the logistic loss. We prove our result on a fixeddataset, both for sampling with or without replacement. Furthermore, forlogistic loss (and similar exponentially-tailed losses), we prove that with SGDthe weight vector converges in direction to the $L_2$ max margin vector as$O(1/\log(t))$ for almost all separable datasets, and the loss converges as$O(1/t)$ - similarly to gradient descent. Lastly, we examine the case of afixed learning rate proportional to the minibatch size. We prove that in thiscase, the asymptotic convergence rate of SGD (with replacement) does not dependon the minibatch size in terms of epochs, if the support vectors span the data.These results may suggest an explanation to similar behaviors observed in deepnetworks, when trained with SGD.

Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate