English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Traditional analyses in non-convex optimization typically rely on thesmoothness assumption, namely requiring the gradients to be Lipschitz. However,recent evidence shows that this smoothness condition does not capture theproperties of some deep learning objective functions, including the onesinvolving Recurrent Neural Networks and LSTMs. Instead, they satisfy a muchmore relaxed condition, with potentially unbounded smoothness. Under thisrelaxed assumption, it has been theoretically and empirically shown that thegradient-clipped SGD has an advantage over the vanilla one. In this paper, weshow that clipping is not indispensable for Adam-type algorithms in tacklingsuch scenarios: we theoretically prove that a generalized SignSGD algorithm canobtain similar convergence rates as SGD with clipping but does not needexplicit clipping at all. This family of algorithms on one end recovers SignSGDand on the other end closely resembles the popular Adam algorithm. Our analysisunderlines the critical role that momentum plays in analyzing SignSGD-type andAdam-type algorithms: it not only reduces the effects of noise, thus removingthe need for large mini-batch in previous analyses of SignSGD-type algorithms,but it also substantially reduces the effects of unbounded smoothness andgradient norms. We also compare these algorithms with popular optimizers on aset of deep learning tasks, observing that we can match the performance of Adamwhile beating the others.

Robustness to Unbounded Smoothness of Generalized SignSGD