English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We propose \emph{Taylorized training} as an initiative towards betterunderstanding neural network training at finite width. Taylorized traininginvolves training the $k$-th order Taylor expansion of the neural network atinitialization, and is a principled extension of linearized training---arecently proposed theory for understanding the success of deep learning.  We experiment with Taylorized training on modern neural networkarchitectures, and show that Taylorized training (1) agrees with full neuralnetwork training increasingly better as we increase $k$, and (2) cansignificantly close the performance gap between linearized and full training.Compared with linearized training, higher-order training works in morerealistic settings such as standard parameterization and large (initial)learning rate. We complement our experiments with theoretical results showingthat the approximation error of $k$-th order Taylorized models decayexponentially over $k$ in wide neural networks.

Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Width