English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Previous work has shown that DNNs with large depth $L$ and$L_{2}$-regularization are biased towards learning low-dimensionalrepresentations of the inputs, which can be interpreted as minimizing a notionof rank $R^{(0)}(f)$ of the learned function $f$, conjectured to be theBottleneck rank. We compute finite depth corrections to this result, revealinga measure $R^{(1)}$ of regularity which bounds the pseudo-determinant of theJacobian $\left|Jf(x)\right|_{+}$ and is subadditive under composition andaddition. This formalizes a balance between learning low-dimensionalrepresentations and minimizing complexity/irregularity in the feature maps,allowing the network to learn the `right' inner dimension. Finally, we provethe conjectured bottleneck structure in the learned features as $L\to\infty$:for large depths, almost all hidden representations are approximately$R^{(0)}(f)$-dimensional, and almost all weight matrices $W_{\ell}$ have$R^{(0)}(f)$ singular values close to 1 while the others are$O(L^{-\frac{1}{2}})$. Interestingly, the use of large learning rates isrequired to guarantee an order $O(L)$ NTK which in turns guarantees infinitedepth convergence of the representations of almost all layers.

Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoff