English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Neural network training is usually accomplished by solving a non-convexoptimization problem using stochastic gradient descent. Although one optimizesover the networks parameters, the main loss function generally only depends onthe realization of the neural network, i.e. the function it computes. Studyingthe optimization problem over the space of realizations opens up new ways tounderstand neural network training. In particular, usual loss functions likemean squared error and categorical cross entropy are convex on spaces of neuralnetwork realizations, which themselves are non-convex. Approximationcapabilities of neural networks can be used to deal with the latternon-convexity, which allows us to establish that for sufficiently largenetworks local minima of a regularized optimization problem on the realizationspace are almost optimal. Note, however, that each realization has manydifferent, possibly degenerate, parametrizations. In particular, a localminimum in the parametrization space needs not correspond to a local minimum inthe realization space. To establish such a connection, inverse stability of therealization map is required, meaning that proximity of realizations must implyproximity of corresponding parametrizations. We present pathologies whichprevent inverse stability in general, and, for shallow networks, proceed toestablish a restricted space of parametrizations on which we have inversestability w.r.t. to a Sobolev norm. Furthermore, we show that by optimizingover such restricted sets, it is still possible to learn any function which canbe learned by optimization over unrestricted sets.

How degenerate is the parametrization of neural networks with the ReLU activation function?