English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

We discover restrained numerical instabilities in current training practicesof deep networks with stochastic gradient descent (SGD), and its variants. Weshow numerical error (on the order of the smallest floating point bit and thusthe most extreme or limiting numerical perturbations induced from floatingpoint arithmetic in training deep nets can be amplified significantly andresult in significant test accuracy variance (sensitivities), comparable to thetest accuracy variance due to stochasticity in SGD. We show how this is likelytraced to instabilities of the optimization dynamics that are restrained, i.e.,localized over iterations and regions of the weight tensor space. We do this bypresenting a theoretical framework using numerical analysis of partialdifferential equations (PDE), and analyzing the gradient descent PDE ofconvolutional neural networks (CNNs). We show that it is stable only undercertain conditions on the learning rate and weight decay. We show that ratherthan blowing up when the conditions are violated, the instability can berestrained. We show this is a consequence of the non-linear PDE associated withthe gradient descent of the CNN, whose local linearization changes whenover-driving the step size of the discretization, resulting in a stabilizingeffect. We link restrained instabilities to the recently discovered Edge ofStability (EoS) phenomena, in which the stable step size predicted by classicaltheory is exceeded while continuing to optimize the loss and still converging.Because restrained instabilities occur at the EoS, our theory provides newinsights and predictions about the EoS, in particular, the role ofregularization and the dependence on the network complexity.

A PDE-based Explanation of Extreme Numerical Sensitivities and Edge of Stability in Training Neural Networks