English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The convergence of GD and SGD when training mildly parameterized neuralnetworks starting from random initialization is studied. For a broad range ofmodels and loss functions, including the most commonly used square loss andcross entropy loss, we prove an ``early stage convergence'' result. We showthat the loss is decreased by a significant amount in the early stage of thetraining, and this decrease is fast. Furthurmore, for exponential type lossfunctions, and under some assumptions on the training data, we show globalconvergence of GD. Instead of relying on extreme over-parameterization, ourstudy is based on a microscopic analysis of the activation patterns for theneurons, which helps us derive more powerful lower bounds for the gradient. Theresults on activation patterns, which we call ``neuron partition'', help buildintuitions for understanding the behavior of neural networks' trainingdynamics, and may be of independent interest.

Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networks