English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Adam-type methods, the extension of adaptive gradient methods, have showngreat performance in the training of both supervised and unsupervised machinelearning models. In particular, Adam-type optimizers have been widely usedempirically as the default tool for training generative adversarial networks(GANs). On the theory side, however, despite the existence of theoreticalresults showing the efficiency of Adam-type methods in minimization problems,the reason of their wonderful performance still remains absent in GAN'straining. In existing works, the fast convergence has long been considered asone of the most important reasons and multiple works have been proposed to givea theoretical guarantee of the convergence to a critical point of min-maxoptimization algorithms under certain assumptions. In this paper, we firstlyargue empirically that in GAN's training, Adam does not converge to a criticalpoint even upon successful training: Only the generator is converging while thediscriminator's gradient norm remains high throughout the training. We namethis one-sided convergence. Then we bridge the gap between experiments andtheory by showing that Adam-type algorithms provably converge to a one-sidedfirst order stationary points in min-max optimization problems under theone-sided MVI condition. We also empirically verify that such one-sided MVIcondition is satisfied for standard GANs after trained over standard data sets.To the best of our knowledge, this is the very first result which provides anempirical observation and a strict theoretical guarantee on the one-sidedconvergence of Adam-type algorithms in min-max optimization.

On the One-sided Convergence of Adam-type Algorithms in Non-convex Non-concave Min-max Optimization