English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We propose a tuning-free dynamic SGD step size formula, which we callDistance over Gradients (DoG). The DoG step sizes depend on simple empiricalquantities (distance from the initial point and norms of gradients) and have no``learning rate'' parameter. Theoretically, we show that a slight variation ofthe DoG formula enjoys strong parameter-free convergence guarantees forstochastic convex optimization assuming only \emph{locally bounded} stochasticgradients. Empirically, we consider a broad range of vision and languagetransfer learning tasks, and show that DoG's performance is close to that ofSGD with tuned learning rate. We also propose a per-layer variant of DoG thatgenerally outperforms tuned SGD, approaching the performance of tuned Adam. APyTorch implementation is available at https://github.com/formll/dog

DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedule