English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

The rise of deep learning in recent years has brought with it increasinglyclever optimization methods to deal with complex, non-linear loss functions.These methods are often designed with convex optimization in mind, but havebeen shown to work well in practice even for the highly non-convex optimizationassociated with neural networks. However, one significant drawback of thesemethods when they are applied to deep learning is that the magnitude of theupdate step is sometimes disproportionate to the magnitude of the weights (muchsmaller or larger), leading to training instabilities such as vanishing andexploding gradients. An idea to combat this issue is gradient descent withproportional updates. Gradient descent with proportional updates was introducedin 2017. It was independently developed by You et al (Layer-wise Adaptive RateScaling (LARS) algorithm) and by Abu-El-Haija (PercentDelta algorithm). Thebasic idea of both of these algorithms is to make each step of the gradientdescent proportional to the current weight norm and independent of the gradientmagnitude. It is common in the context of new optimization methods to proveconvergence or derive regret bounds under the assumption of Lipschitzcontinuity and convexity. However, even though LARS and PercentDelta were shownto work well in practice, there is no theoretical analysis of the convergenceproperties of these algorithms. Thus it is not clear if the idea of gradientdescent with proportional updates is used in the optimal way, or if it could beimproved by using a different norm or specific learning rate schedule, forexample. Moreover, it is not clear if these algorithms can be extended to otherproblems, besides neural networks. We attempt to answer these questions byestablishing the theoretical analysis of gradient descent with proportionalupdates, and verifying this analysis with empirical examples.

Convergence Analysis of Gradient Descent Algorithms with Proportional Updates