English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Modern deep learning (DL) architectures are trained using variants of the SGDalgorithm that is run with a $\textit{manually}$ defined learning rateschedule, i.e., the learning rate is dropped at the pre-defined epochs,typically when the training loss is expected to saturate. In this paper wedevelop an algorithm that realizes the learning rate drop$\textit{automatically}$. The proposed method, that we refer to as AutoDrop, ismotivated by the observation that the angular velocity of the model parameters,i.e., the velocity of the changes of the convergence direction, for a fixedlearning rate initially increases rapidly and then progresses towards softsaturation. At saturation the optimizer slows down thus the angular velocitysaturation is a good indicator for dropping the learning rate. After the drop,the angular velocity "resets" and follows the previously described pattern - itincreases again until saturation. We show that our method improves over SOTAtraining approaches: it accelerates the training of DL models and leads to abetter generalization. We also show that our method does not require any extrahyperparameter tuning. AutoDrop is furthermore extremely simple to implementand computationally cheap. Finally, we develop a theoretical framework foranalyzing our algorithm and provide convergence guarantees.

AutoDrop: Training Deep Learning Models with Automatic Learning Rate Drop