English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Stochastic gradient descent (SGD) and adaptive gradient methods, such as Adamand RMSProp, have been widely used in training deep neural networks. Weempirically show that while the difference between the standard generalizationperformance of models trained using these methods is small, those trained usingSGD exhibit far greater robustness under input perturbations. Notably, ourinvestigation demonstrates the presence of irrelevant frequencies in naturaldatasets, where alterations do not affect models' generalization performance.However, models trained with adaptive methods show sensitivity to thesechanges, suggesting that their use of irrelevant frequencies can lead tosolutions sensitive to perturbations. To better understand this difference, westudy the learning dynamics of gradient descent (GD) and sign gradient descent(signGD) on a synthetic dataset that mirrors natural signals. With athree-dimensional input space, the models optimized with GD and signGD havestandard risks close to zero but vary in their adversarial risks. Our resultshows that linear models' robustness to $\ell_2$-norm bounded changes isinversely proportional to the model parameters' weight norm: a smaller weightnorm implies better robustness. In the context of deep learning, ourexperiments show that SGD-trained neural networks have smaller Lipschitzconstants, explaining the better robustness to input perturbations than thosetrained with adaptive gradient methods.

Understanding the robustness difference between stochastic gradient descent and adaptive gradient methods