English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Despite their overwhelming capacity to overfit, deep neural networks trainedby specific optimization algorithms tend to generalize well to unseen data.Recently, researchers explained it by investigating the implicit regularizationeffect of optimization algorithms. A remarkable progress is the work (Lyu&Li,2019), which proves gradient descent (GD) maximizes the margin of homogeneousdeep neural networks. Except GD, adaptive algorithms such as AdaGrad, RMSPropand Adam are popular owing to their rapid training process. However,theoretical guarantee for the generalization of adaptive optimizationalgorithms is still lacking. In this paper, we study the implicitregularization of adaptive optimization algorithms when they are optimizing thelogistic loss on homogeneous deep neural networks. We prove that adaptivealgorithms that adopt exponential moving average strategy in conditioner (suchas Adam and RMSProp) can maximize the margin of the neural network, whileAdaGrad that directly sums historical squared gradients in conditioner can not.It indicates superiority on generalization of exponential moving averagestrategy in the design of the conditioner. Technically, we provide a unifiedframework to analyze convergent direction of adaptive optimization algorithmsby constructing novel adaptive gradient flow and surrogate margin. Ourexperiments can well support the theoretical findings on convergent directionof adaptive optimization algorithms.

The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks