English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

While stochastic gradient descent (SGD) is still the most popularoptimization algorithm in deep learning, adaptive algorithms such as Adam haveestablished empirical advantages over SGD in some deep learning applicationssuch as training transformers. However, it remains a question that why Adamconverges significantly faster than SGD in these scenarios. In this paper, wepropose one explanation of why Adam converges faster than SGD using a newconcept directional sharpness. We argue that the performance of optimizationalgorithms is closely related to the directional sharpness of the update steps,and show SGD has much worse directional sharpness compared to adaptivealgorithms. We further observe that only a small fraction of the coordinatescauses the bad sharpness and slow convergence of SGD, and propose to usecoordinate-wise clipping as a solution to SGD and other optimizationalgorithms. We demonstrate the effect of coordinate-wise clipping on sharpnessreduction and speeding up the convergence of optimization algorithms undervarious settings. We show that coordinate-wise clipping improves the local lossreduction when only a small fraction of the coordinates has bad sharpness. Weconclude that the sharpness reduction effect of adaptive coordinate-wisescaling is the reason for Adam's success in practice and suggest the use ofcoordinate-wise clipping as a universal technique to speed up deep learningoptimization.

Toward Understanding Why Adam Converges Faster Than SGD for Transformers