English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

We study training one-hidden-layer ReLU networks in the neural tangent kernel(NTK) regime, where the networks' biases are initialized to some constantrather than zero. We prove that under such initialization, the neural networkwill have sparse activation throughout the entire training process, whichenables fast training procedures via some sophisticated computational methods.With such initialization, we show that the neural networks possess a differentlimiting kernel which we call \textit{bias-generalized NTK}, and we studyvarious properties of the neural networks with this new kernel. We firstcharacterize the gradient descent dynamics. In particular, we show that thenetwork in this case can achieve as fast convergence as the dense network, asopposed to the previous work suggesting that the sparse networks convergeslower. In addition, our result improves the previous required width to ensureconvergence. Secondly, we study the networks' generalization: we show awidth-sparsity dependence, which yields a sparsity-dependent Rademachercomplexity and generalization bound. To our knowledge, this is the firstsparsity-dependent generalization result via Rademacher complexity. Lastly, westudy the smallest eigenvalue of this new kernel. We identify a data-dependentregion where we can derive a much sharper lower bound on the NTK's smallesteigenvalue than the worst-case bound previously known. This can lead toimprovement in the generalization bound.

Neural Networks with Sparse Activation Induced by Large Bias: Tighter Analysis with Bias-Generalized NTK