English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

While Standard gradient descent is one very popular optimisation method, itsconvergence cannot be proven beyond the class of functions whose gradient isglobally Lipschitz continuous. As such, it is not actually applicable torealistic applications such as Deep Neural Networks. In this paper, we provethat its backtracking variant behaves very nicely, in particular convergencecan be shown for all Morse functions. The main theoretical result of this paperis as follows.  Theorem. Let $f:\mathbb{R}^k\rightarrow \mathbb{R}$ be a $C^1$ function, and$\{z_n\}$ a sequence constructed from the Backtracking gradient descentalgorithm. (1) Either $\lim _{n\rightarrow\infty}||z_n||=\infty$ or $\lim_{n\rightarrow\infty}||z_{n+1}-z_n||=0$. (2) Assume that $f$ has at mostcountably many critical points. Then either $\lim_{n\rightarrow\infty}||z_n||=\infty$ or $\{z_n\}$ converges to a critical pointof $f$. (3) More generally, assume that all connected components of the set ofcritical points of $f$ are compact. Then either $\lim_{n\rightarrow\infty}||z_n||=\infty$ or $\{z_n\}$ is bounded. Moreover, in thelatter case the set of cluster points of $\{z_n\}$ is connected.  Some generalised versions of this result, including an inexact version, areincluded. Another result in this paper concerns the problem of saddle points.We then present a heuristic argument to explain why Standard gradient descentmethod works so well, and modifications of the backtracking versions of GD, MMTand NAG. Experiments with datasets CIFAR10 and CIFAR100 on various populararchitectures verify the heuristic argument also for the mini-batch practiceand show that our new algorithms, while automatically fine tuning learningrates, perform better than current state-of-the-art methods such as MMT, NAG,Adagrad, Adadelta, RMSProp, Adam and Adamax.

Backtracking gradient descent method for general $C^1$ functions, with applications to Deep Learning