English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We study the implicit regularization imposed by gradient descent for learningmulti-layer homogeneous functions including feed-forward fully connected andconvolutional deep neural networks with linear, ReLU or Leaky ReLU activation.We rigorously prove that gradient flow (i.e. gradient descent withinfinitesimal step size) effectively enforces the differences between squarednorms across different layers to remain invariant without any explicitregularization. This result implies that if the weights are initially small,gradient flow automatically balances the magnitudes of all layers. Using adiscretization argument, we analyze gradient descent with positive step sizefor the non-convex low-rank asymmetric matrix factorization problem without anyregularization. Inspired by our findings for gradient flow, we prove thatgradient descent with step sizes $\eta_t = O\left(t^{-\left(\frac12+\delta\right)} \right)$ ($0<\delta\le\frac12$) automatically balancestwo low-rank factors and converges to a bounded global optimum. Furthermore,for rank-$1$ asymmetric matrix factorization we give a finer analysis showinggradient descent with constant step size converges to the global minimum at aglobally linear rate. We believe that the idea of examining the invarianceimposed by first order algorithms in learning homogeneous models could serve asa fundamental building block for studying optimization for learning deepmodels.

Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced