English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The residual network is now one of the most effective structures in deeplearning, which utilizes the skip connections to ``guarantee" the performancewill not get worse. However, the non-convexity of the neural network makes itunclear whether the skip connections do provably improve the learning abilitysince the nonlinearity may create many local minima. In some previous works\cite{freeman2016topology}, it is shown that despite the non-convexity, theloss landscape of the two-layer ReLU network has good properties when thenumber $m$ of hidden nodes is very large. In this paper, we follow this line tostudy the topology (sub-level sets) of the loss landscape of deep ReLU neuralnetworks with a skip connection and theoretically prove that the skipconnection network inherits the good properties of the two-layer network andskip connections can help to control the connectedness of the sub-level sets,such that any local minima worse than the global minima of some two-layer ReLUnetwork will be very ``shallow". The ``depth" of these local minima are at most$O(m^{(\eta-1)/n})$, where $n$ is the input dimension, $\eta<1$. This providesa theoretical explanation for the effectiveness of the skip connection in deeplearning.

Is the Skip Connection Provable to Reform the Neural Network Loss Landscape?