English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent results in the literature indicate that a residual network (ResNet)composed of a single residual block outperforms linear predictors, in the sensethat all local minima in its optimization landscape are at least as good as thebest linear predictor. However, these results are limited to a single residualblock (i.e., shallow ResNets), instead of the deep ResNets composed of multipleresidual blocks. We take a step towards extending this result to deep ResNets.We start by two motivating examples. First, we show that there exist datasetsfor which all local minima of a fully-connected ReLU network are no better thanthe best linear predictor, whereas a ResNet has strictly better local minima.Second, we show that even at the global minimum, the representation obtainedfrom the residual block outputs of a 2-block ResNet do not necessarily improvemonotonically over subsequent blocks, which highlights a fundamental difficultyin analyzing deep ResNets. Our main theorem on deep ResNets shows under simplegeometric conditions that, any critical point in the optimization landscape iseither (i) at least as good as the best linear predictor; or (ii) the Hessianat this critical point has a strictly negative eigenvalue. Notably, our theoremshows that a chain of multiple skip-connections can improve the optimizationlandscape, whereas existing results study direct skip-connections to the lasthidden layer or output layer. Finally, we complement our results by showingbenign properties of the "near-identity regions" of deep ResNets, showingdepth-independent upper bounds for the risk attained at critical points as wellas the Rademacher complexity.

Are deep ResNets provably better than linear predictors?