English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Gradient descent (GD) type optimization schemes are the standard methods totrain artificial neural networks (ANNs) with rectified linear unit (ReLU)activation. Such schemes can be considered as discretizations of gradient flows(GFs) associated to the training of ANNs with ReLU activation and most of thekey difficulties in the mathematical convergence analysis of GD typeoptimization schemes in the training of ANNs with ReLU activation seem to bealready present in the dynamics of the corresponding GF differential equations.It is the key subject of this work to analyze such GF differential equations inthe training of ANNs with ReLU activation and three layers (one input layer,one hidden layer, and one output layer). In particular, in this article weprove in the case where the target function is possibly multi-dimensional andcontinuous and in the case where the probability distribution of the input datais absolutely continuous with respect to the Lebesgue measure that the risk ofevery bounded GF trajectory converges to the risk of a critical point. Inaddition, in this article we show in the case of a 1-dimensional affine lineartarget function and in the case where the probability distribution of the inputdata coincides with the standard uniform distribution that the risk of everybounded GF trajectory converges to zero if the initial risk is sufficientlysmall. Finally, in the special situation where there is only one neuron on thehidden layer (1-dimensional hidden layer) we strengthen the above named resultfor affine linear target functions by proving that that the risk of every (notnecessarily bounded) GF trajectory converges to zero if the initial risk issufficiently small.

Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation