English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We study the effect of mini-batching on the loss landscape of deep neuralnetworks using spiked, field-dependent random matrix theory. We demonstratethat the magnitude of the extremal values of the batch Hessian are larger thanthose of the empirical Hessian. We also derive similar results for theGeneralised Gauss-Newton matrix approximation of the Hessian. As a consequenceof our theorems we derive an analytical expressions for the maximal learningrates as a function of batch size, informing practical training regimens forboth stochastic gradient descent (linear scaling) and adaptive algorithms, suchas Adam (square root scaling), for smooth, non-convex deep neural networks.Whilst the linear scaling for stochastic gradient descent has been derivedunder more restrictive conditions, which we generalise, the square root scalingrule for adaptive optimisers is, to our knowledge, completely novel. %Forstochastic second-order methods and adaptive methods, we derive that theminimal damping coefficient is proportional to the ratio of the learning rateto batch size. We validate our claims on the VGG/WideResNet architectures onthe CIFAR-$100$ and ImageNet datasets. Based on our investigations of thesub-sampled Hessian we develop a stochastic Lanczos quadrature based on the flylearning rate and momentum learner, which avoids the need for expensivemultiple evaluations for these key hyper-parameters and shows good preliminaryresults on the Pre-Residual Architecure for CIFAR-$100$.

Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training