English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Models recently used in the literature proving residual networks (ResNets)are better than linear predictors are actually different from standard ResNetsthat have been widely used in computer vision. In addition to the assumptionssuch as scalar-valued output or single residual block, these models have nononlinearities at the final residual representation that feeds into the finalaffine layer. To codify such a difference in nonlinearities and reveal a linearestimation property, we define ResNEsts, i.e., Residual Nonlinear Estimators,by simply dropping nonlinearities at the last residual representation fromstandard ResNets. We show that wide ResNEsts with bottleneck blocks can alwaysguarantee a very desirable training property that standard ResNets aim toachieve, i.e., adding more blocks does not decrease performance given the sameset of basis elements. To prove that, we first recognize ResNEsts are basisfunction models that are limited by a coupling problem in basis learning andlinear prediction. Then, to decouple prediction weights from basis learning, weconstruct a special architecture termed augmented ResNEst (A-ResNEst) thatalways guarantees no worse performance with the addition of a block. As aresult, such an A-ResNEst establishes empirical risk lower bounds for a ResNEstusing corresponding bases. Our results demonstrate ResNEsts indeed have aproblem of diminishing feature reuse; however, it can be avoided bysufficiently expanding or widening the input space, leading to theabove-mentioned desirable property. Inspired by the DenseNets that have beenshown to outperform ResNets, we also propose a corresponding new model calledDensely connected Nonlinear Estimator (DenseNEst). We show that any DenseNEstcan be represented as a wide ResNEst with bottleneck blocks. Unlike ResNEsts,DenseNEsts exhibit the desirable property without any special architecturalre-design.

ResNEsts and DenseNEsts: Block-based DNN Models with Improved Representation Guarantees