English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

This paper studies how neural network architecture affects the speed oftraining. We introduce a simple concept called gradient confusion to helpformally analyze this. When gradient confusion is high, stochastic gradientsproduced by different data samples may be negatively correlated, slowing downconvergence. But when gradient confusion is low, data samples interactharmoniously, and training proceeds quickly. Through theoretical andexperimental results, we demonstrate how the neural network architectureaffects gradient confusion, and thus the efficiency of training. Our resultsshow that, for popular initialization techniques, increasing the width ofneural networks leads to lower gradient confusion, and thus faster modeltraining. On the other hand, increasing the depth of neural networks has theopposite effect. Our results indicate that alternate initialization techniquesor networks using both batch normalization and skip connections help reduce thetraining burden of very deep networks.

The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent