English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Understanding the learning dynamics and inductive bias of neural networks(NNs) is hindered by the opacity of the relationship between NN parameters andthe function represented. We propose reparametrizing ReLU NNs as continuouspiecewise linear splines. Using this spline lens, we study learning dynamics inshallow univariate ReLU NNs, finding unexpected insights and explanations forseveral perplexing phenomena. We develop a surprisingly simple and transparentview of the structure of the loss surface, including its critical and fixedpoints, Hessian, and Hessian spectrum. We also show that standard weightinitializations yield very flat functions, and that this flatness, togetherwith overparametrization and the initial weight scale, is responsible for thestrength and type of implicit regularization, consistent with recent workarXiv:1906.05827. Our implicit regularization results are complementary torecent work arXiv:1906.07842, done independently, which showed thatinitialization scale critically controls implicit regularization via akernel-based argument. Our spline-based approach reproduces their key implicitregularization results but in a far more intuitive and transparent manner.Going forward, our spline-based approach is likely to extend naturally to themultivariate and deep settings, and will play a foundational role in efforts tounderstand neural networks. Videos of learning dynamics using a spline-basedvisualization are available at http://shorturl.at/tFWZ2.

Shallow Univariate ReLu Networks as Splines: Initialization, Loss Surface, Hessian, & Gradient Flow Dynamics