English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this work, we consider the optimization process of minibatch stochasticgradient descent (SGD) on a 2-layer neural network with data separated by aquadratic ground truth function. We prove that with data drawn from the$d$-dimensional Boolean hypercube labeled by the quadratic ``XOR'' function $y= -x_ix_j$, it is possible to train to a population error $o(1)$ with $d\:\text{polylog}(d)$ samples. Our result considers simultaneously training bothlayers of the two-layer-neural network with ReLU activations via standardminibatch SGD on the logistic loss. To our knowledge, this work is the first togive a sample complexity of $\tilde{O}(d)$ for efficiently learning the XORfunction on isotropic data on a standard neural network with standard training.Our main technique is showing that the network evolves in two phases: a$\textit{signal-finding}$ phase where the network is small and many of theneurons evolve independently to find features, and a $\textit{signal-heavy}$phase, where SGD maintains and balances the features. We leverage thesimultaneous training of the layers to show that it is sufficient for only asmall fraction of the neurons to learn features, since those neurons will beamplified by the simultaneous growth of their second layer weights.

SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problem