English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Recently, significant progress has been made in understanding thegeneralization of neural networks (NNs) trained by gradient descent (GD) usingthe algorithmic stability approach. However, most of the existing research hasfocused on one-hidden-layer NNs and has not addressed the impact of differentnetwork scaling parameters. In this paper, we greatly extend the previous work\cite{lei2022stability,richards2021stability} by conducting a comprehensivestability and generalization analysis of GD for multi-layer NNs. For two-layerNNs, our results are established under general network scaling parameters,relaxing previous conditions. In the case of three-layer NNs, our technicalcontribution lies in demonstrating its nearly co-coercive property by utilizinga novel induction strategy that thoroughly explores the effects ofover-parameterization. As a direct application of our general findings, wederive the excess risk rate of $O(1/\sqrt{n})$ for GD algorithms in bothtwo-layer and three-layer NNs. This sheds light on sufficient or necessaryconditions for under-parameterized and over-parameterized NNs trained by GD toattain the desired risk rate of $O(1/\sqrt{n})$. Moreover, we demonstrate thatas the scaling parameter increases or the network complexity decreases, lessover-parameterization is required for GD to achieve the desired error rates.Additionally, under a low-noise condition, we obtain a fast risk rate of$O(1/n)$ for GD in both two-layer and three-layer NNs.

Generalization Guarantees of Gradient Descent for Multi-Layer Neural Networks