English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Over-parametrization has become a popular technique in deep learning. It isobserved that by over-parametrization, a larger neural network needs a fewertraining iterations than a smaller one to achieve a certain level ofperformance -- namely, over-parametrization leads to acceleration inoptimization. However, despite that over-parametrization is widely usednowadays, little theory is available to explain the acceleration due toover-parametrization. In this paper, we propose understanding it by studying asimple problem first. Specifically, we consider the setting that there is asingle teacher neuron with quadratic activation, where over-parametrization isrealized by having multiple student neurons learn the data generated from theteacher neuron. We provably show that over-parametrization helps the iterategenerated by gradient descent to enter the neighborhood of a global optimalsolution that achieves zero testing error faster. On the other hand, we alsopoint out an issue regarding the necessity of over-parametrization and studyhow the scaling of the output neurons affects the convergence time.

Understanding How Over-Parametrization Leads to Acceleration: A case of learning a single teacher neuron