English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

This paper investigates asymptotic behaviors of gradient descent algorithms(particularly accelerated gradient descent and stochastic gradient descent) inthe context of stochastic optimization arising in statistics and machinelearning where objective functions are estimated from available data. We showthat these algorithms can be computationally modeled by continuous-timeordinary or stochastic differential equations. We establish gradient flowcentral limit theorems to describe the limiting dynamic behaviors of thesecomputational algorithms and the large-sample performances of the relatedstatistical procedures, as the number of algorithm iterations and data sizeboth go to infinity, where the gradient flow central limit theorems aregoverned by some linear ordinary or stochastic differential equations liketime-dependent Ornstein-Uhlenbeck processes. We illustrate that our study canprovide a novel unified framework for a joint computational and statisticalasymptotic analysis, where the computational asymptotic analysis studiesdynamic behaviors of these algorithms with the time (or the number ofiterations in the algorithms), the statistical asymptotic analysis investigateslarge sample behaviors of the statistical procedures (like estimators andclassifiers) that the algorithms are applied to compute, and in fact thestatistical procedures are equal to the limits of the random sequencesgenerated from these iterative algorithms as the number of iterations goes toinfinity. The joint analysis results based on the obtained gradient flowcentral limit theorems can identify four factors - learning rate, batch size,gradient covariance, and Hessian - to derive new theory regarding the localminima found by stochastic gradient descent for solving non-convex optimizationproblems.

Asymptotic Analysis via Stochastic Differential Equations of Gradient Descent Algorithms in Statistical and Computational Paradigms