English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent works have shown that on sufficiently over-parametrized neural nets,gradient descent with relatively large initialization optimizes a predictionfunction in the RKHS of the Neural Tangent Kernel (NTK). This analysis leads toglobal convergence results but does not work when there is a standard $\ell_2$regularizer, which is useful to have in practice. We show that sampleefficiency can indeed depend on the presence of the regularizer: we construct asimple distribution in d dimensions which the optimal regularized neural netlearns with $O(d)$ samples but the NTK requires $\Omega(d^2)$ samples to learn.To prove this, we establish two analysis tools: i) for multi-layer feedforwardReLU nets, we show that the global minimizer of a weakly-regularizedcross-entropy loss is the max normalized margin solution among all neural nets,which generalizes well; ii) we develop a new technique for proving lower boundsfor kernel methods, which relies on showing that the kernel cannot focus oninformative features. Motivated by our generalization results, we study whetherthe regularized global optimum is attainable. We prove that for infinite-widthtwo-layer nets, noisy gradient descent optimizes the regularized neural netloss to a global minimum in polynomial iterations.

Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel