English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

It is currently known how to characterize functions that neural networks canlearn with SGD for two extremal parameterizations: neural networks in thelinear regime, and neural networks with no structural constraints. However, forthe main parametrization of interest (non-linear but regular networks) no tightcharacterization has yet been achieved, despite significant developments.  We take a step in this direction by considering depth-2 neural networkstrained by SGD in the mean-field regime. We consider functions on binary inputsthat depend on a latent low-dimensional subspace (i.e., small number ofcoordinates). This regime is of interest since it is poorly understood howneural networks routinely tackle high-dimensional datasets and adapt to latentlow-dimensional structure without suffering from the curse of dimensionality.Accordingly, we study SGD-learnability with $O(d)$ sample complexity in a largeambient dimension $d$.  Our main results characterize a hierarchical property, the "merged-staircaseproperty", that is both necessary and nearly sufficient for learning in thissetting.  We further show that non-linear training is necessary: for this class offunctions, linear methods on any feature map (e.g., the NTK) are not capable oflearning efficiently. The key tools are a new "dimension-free" dynamicsapproximation result that applies to functions defined on a latent space oflow-dimension, a proof of global convergence based on polynomial identitytesting, and an improvement of lower bounds against linear methods fornon-almost orthogonal functions.

The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networks