English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Stochastic gradient descent plays a fundamental role in nearly allapplications of deep learning. However its ability to converge to a globalminimum remains shrouded in mystery. In this paper we propose to study thebehavior of the loss function on fixed mini-batches along SGD trajectories. Weshow that the loss function on a fixed batch appears to be remarkablyconvex-like. In particular for ResNet the loss for any fixed mini-batch can beaccurately modeled by a quadratic function and a very low loss value can bereached in just one step of gradient descent with sufficiently large learningrate. We propose a simple model that allows to analyze the relationship betweenthe gradients of stochastic mini-batches and the full batch. Our analysisallows us to discover the equivalency between iterate aggregates and specificlearning rate schedules. In particular, for Exponential Moving Average (EMA)and Stochastic Weight Averaging we show that our proposed model matches theobserved training trajectories on ImageNet. Our theoretical model predicts thatan even simpler averaging technique, averaging just two points a many stepsapart, significantly improves accuracy compared to the baseline. We validatedour findings on ImageNet and other datasets using ResNet architecture.

Training trajectories, mini-batch losses and the curious role of the learning rate