English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Stochastic Gradient Descent (SGD) is an out-of-equilibrium algorithm usedextensively to train artificial neural networks. However very little is knownon to what extent SGD is crucial for to the success of this technology and, inparticular, how much it is effective in optimizing high-dimensional non-convexcost functions as compared to other optimization algorithms such as GradientDescent (GD). In this work we leverage dynamical mean field theory to benchmarkits performances in the high-dimensional limit. To do that, we consider theproblem of recovering a hidden high-dimensional non-linearly encrypted signal,a prototype high-dimensional non-convex hard optimization problem. We comparethe performances of SGD to GD and we show that SGD largely outperforms GD forsufficiently small batch sizes. In particular, a power law fit of therelaxation time of these algorithms shows that the recovery threshold for SGDwith small batch size is smaller than the corresponding one of GD.

Stochastic Gradient Descent outperforms Gradient Descent in recovering a high-dimensional signal in a glassy energy landscape