English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Rectified Linear Units (ReLU) have become the main model for the neural unitsin current deep learning systems. This choice has been originally suggested asa way to compensate for the so called vanishing gradient problem which canundercut stochastic gradient descent (SGD) learning in networks composed ofmultiple layers. Here we provide analytical results on the effects of ReLUs onthe capacity and on the geometrical landscape of the solution space intwo-layer neural networks with either binary or real-valued weights. We studythe problem of storing an extensive number of random patterns and find that,quite unexpectedly, the capacity of the network remains finite as the number ofneurons in the hidden layer increases, at odds with the case of threshold unitsin which the capacity diverges. Possibly more important, a large deviationapproach allows us to find that the geometrical landscape of the solution spacehas a peculiar structure: while the majority of solutions are close in distancebut still isolated, there exist rare regions of solutions which are much moredense than the similar ones in the case of threshold units. These solutions arerobust to perturbations of the weights and can tolerate large perturbations ofthe inputs. The analytical results are corroborated by numerical findings.

Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations