English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Stochastic gradient descent (SGD) is the simplest deep learning optimizerwith which to train deep neural networks. While SGD can use various learningrates, such as constant or diminishing rates, the previous numerical resultsshowed that SGD performs better than other deep learning optimizers using whenit uses learning rates given by line search methods. In this paper, we performa convergence analysis on SGD with a learning rate given by an Armijo linesearch for nonconvex optimization. The analysis indicates that the upper boundof the expectation of the squared norm of the full gradient becomes small whenthe number of steps and the batch size are large. Next, we show that, for SGDwith the Armijo-line-search learning rate, the number of steps needed fornonconvex optimization is a monotone decreasing convex function of the batchsize; that is, the number of steps needed for nonconvex optimization decreasesas the batch size increases. Furthermore, we show that the stochasticfirst-order oracle (SFO) complexity, which is the stochastic gradientcomputation cost, is a convex function of the batch size; that is, there existsa critical batch size that minimizes the SFO complexity. Finally, we providenumerical results that support our theoretical results. The numerical resultsindicate that the number of steps needed for training deep neural networksdecreases as the batch size increases and that there exist the critical batchsizes that can be estimated from the theoretical results.

Relationship between Batch Size and Number of Steps Needed for Nonconvex Optimization of Stochastic Gradient Descent using Armijo Line Search