English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Following early work on Hessian-free methods for deep learning, we study astochastic generalized Gauss-Newton method (SGN) for training DNNs. SGN is asecond-order optimization method, with efficient iterations, that wedemonstrate to often require substantially fewer iterations than standard SGDto converge. As the name suggests, SGN uses a Gauss-Newton approximation forthe Hessian matrix, and, in order to compute an approximate search direction,relies on the conjugate gradient method combined with forward and reverseautomatic differentiation. Despite the success of SGD and its first-ordervariants, and despite Hessian-free methods based on the Gauss-Newton Hessianapproximation having been already theoretically proposed as practical methodsfor training DNNs, we believe that SGN has a lot of undiscovered and yet notfully displayed potential in big mini-batch scenarios. For this setting, wedemonstrate that SGN does not only substantially improve over SGD in terms ofthe number of iterations, but also in terms of runtime. This is made possibleby an efficient, easy-to-use and flexible implementation of SGN we propose inthe Theano deep learning platform, which, unlike Tensorflow and Pytorch,supports forward automatic differentiation. This enables researchers to furtherstudy and improve this promising optimization technique and hopefullyreconsider stochastic second-order methods as competitive optimizationtechniques for training DNNs; we also hope that the promise of SGN may lead toforward automatic differentiation being added to Tensorflow or Pytorch. Ourresults also show that in big mini-batch scenarios SGN is more robust than SGDwith respect to its hyperparameters (we never had to tune its step-size for ourbenchmarks!), which eases the expensive process of hyperparameter tuning thatis instead crucial for the performance of first-order methods.

On the Promise of the Stochastic Generalized Gauss-Newton Method for Training DNNs