English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Pruning neural networks before training has received increasing interest dueto its potential to reduce training time and memory. One popular method is toprune the connections based on a certain metric, but it is not entirely clearwhat metric is the best choice. Recent advances in neural tangent kernel (NTK)theory suggest that the training dynamics of large enough neural networks isclosely related to the spectrum of the NTK. Motivated by this finding, wepropose to prune the connections that have the least influence on the spectrumof the NTK. This method can help maintain the NTK spectrum, which may helpalign the training dynamics to that of its dense counterpart. However, onepossible issue is that the fixed-weight-NTK corresponding to a given initialpoint can be very different from the NTK corresponding to later iterates duringthe training phase. We further propose to sample multiple realizations ofrandom weights to estimate the NTK spectrum. Note that our approach isweight-agnostic, which is different from most existing methods that areweight-dependent. In addition, we use random inputs to compute thefixed-weight-NTK, making our method data-agnostic as well. We name ourforesight pruning algorithm Neural Tangent Kernel Spectrum-Aware Pruning(NTK-SAP). Empirically, our method achieves better performance than allbaselines on multiple datasets.

NTK-SAP: Improving neural network pruning by aligning training dynamics