English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this article we investigate blow up phenomena for gradient descentoptimization methods in the training of artificial neural networks (ANNs). Ourtheoretical analysis is focused on shallow ANNs with one neuron on the inputlayer, one neuron on the output layer, and one hidden layer. For ANNs with ReLUactivation and at least two neurons on the hidden layer we establish theexistence of a target function such that there exists a lower bound for therisk values of the critical points of the associated risk function which isstrictly greater than the infimum of the image of the risk function. Thisallows us to demonstrate that every gradient flow trajectory with an initialrisk smaller than this lower bound diverges. Furthermore, we analyze andcompare various popular types of activation functions with regard to thedivergence of gradient flow trajectories and gradient descent trajectories inthe training of ANNs and with regard to the closely related question concerningthe existence of global minimum points of the risk function.

Blow up phenomena for gradient descent optimization methods in the training of artificial neural networks