English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently, convergence as well as convergence rate analyses of deep learningoptimizers for nonconvex optimization have been widely studied. Meanwhile,numerical evaluations for the optimizers have precisely clarified therelationship between batch size and the number of steps needed for trainingdeep neural networks. The main contribution of this paper is to showtheoretically that the number of steps needed for nonconvex optimization ofeach of the optimizers can be expressed as a rational function of batch size.Having these rational functions leads to two particularly important facts,which were validated numerically in previous studies. The first fact is thatthere exists an optimal batch size such that the number of steps needed fornonconvex optimization is minimized. This implies that using larger batch sizesthan the optimal batch size does not decrease the number of steps needed fornonconvex optimization. The second fact is that the optimal batch size dependson the optimizer. In particular, it is shown theoretically that momentum andAdam-type optimizers can exploit larger optimal batches and further reduce theminimum number of steps needed for nonconvex optimization than can thestochastic gradient descent optimizer.

The Number of Steps Needed for Nonconvex Optimization of a Deep Learning Optimizer is a Rational Function of Batch Size