English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We consider non-convex stochastic optimization problems where the objectivefunctions have super-linearly growing and discontinuous stochastic gradients.In such a setting, we provide a non-asymptotic analysis for the tamedunadjusted stochastic Langevin algorithm (TUSLA) introduced in Lovas et al.(2020). In particular, we establish non-asymptotic error bounds for the TUSLAalgorithm in Wasserstein-1 and Wasserstein-2 distances. The latter resultenables us to further derive non-asymptotic estimates for the expected excessrisk. To illustrate the applicability of the main results, we consider anexample from transfer learning with ReLU neural networks, which represents akey paradigm in machine learning. Numerical experiments are presented for theaforementioned example which support our theoretical findings. Hence, in thissetting, we demonstrate both theoretically and numerically that the TUSLAalgorithm can solve the optimization problem involving neural networks withReLU activation function. Besides, we provide simulation results for syntheticexamples where popular algorithms, e.g. ADAM, AMSGrad, RMSProp, and (vanilla)stochastic gradient descent (SGD) algorithm, may fail to find the minimizer ofthe objective functions due to the super-linear growth and the discontinuity ofthe corresponding stochastic gradient, while the TUSLA algorithm convergesrapidly to the optimal solution. Moreover, we provide an empirical comparisonof the performance of TUSLA with popular stochastic optimizers on real-worlddatasets, as well as investigate the effect of the key hyperparameters of TUSLAon its performance.

Non-asymptotic estimates for TUSLA algorithm for non-convex learning with applications to neural networks with ReLU activation function