English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We prove that two-layer (Leaky)ReLU networks initialized by e.g. the widelyused method proposed by He et al. (2015) and trained using gradient descent ona least-squares loss are not universally consistent. Specifically, we describea large class of one-dimensional data-generating distributions for which, withhigh probability, gradient descent only finds a bad local minimum of theoptimization landscape, since it is unable to move the biases far away fromtheir initialization at zero. It turns out that in these cases, the foundnetwork essentially performs linear regression even if the target function isnon-linear. We further provide numerical evidence that this happens inpractical situations, for some multi-dimensional distributions and thatstochastic gradient descent exhibits similar behavior. We also provideempirical results on how the choice of initialization and optimizer caninfluence this behavior.

Training Two-Layer ReLU Networks with Gradient Descent is Inconsistent