English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We show that the representation cost of fully connected neural networks withhomogeneous nonlinearities - which describes the implicit bias in functionspace of networks with $L_2$-regularization or with losses such as thecross-entropy - converges as the depth of the network goes to infinity to anotion of rank over nonlinear functions. We then inquire under which conditionsthe global minima of the loss recover the `true' rank of the data: we show thatfor too large depths the global minimum will be approximately rank 1(underestimating the rank); we then argue that there is a range of depths whichgrows with the number of datapoints where the true rank is recovered. Finally,we discuss the effect of the rank of a classifier on the topology of theresulting class boundaries and show that autoencoders with optimal nonlinearrank are naturally denoising.

Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functions