English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

LARS and LAMB have emerged as prominent techniques in Large Batch Learning(LBL) to ensure training stability in AI. Convergence stability is a challengein LBL, where the AI agent usually gets trapped in the sharp minimizer. Toaddress this challenge, warm-up is an efficient technique, but it lacks astrong theoretical foundation. Specifically, the warm-up process often reducesgradients in the early phase, inadvertently preventing the agent from escapingthe sharp minimizer early on. In light of this situation, we conduct empiricalexperiments to analyze the behaviors of LARS and LAMB with and without awarm-up strategy. Our analyses give a comprehensive insight into the behaviorsof LARS, LAMB, and the necessity of a warm-up technique in LBL, including anexplanation of their failure in many cases. Building upon these insights, wepropose a novel algorithm called Time Varying LARS (TVLARS), which facilitatesrobust training in the initial phase without the need for warm-up. Aconfigurable sigmoid-like function is employed in TVLARS to replace the warm-upprocess to enhance training stability. Moreover, TVLARS stimulates gradientexploration in the early phase, thus allowing it to surpass the sharp minimizesearly on and gradually transition to LARS and achieving robustness of LARS inthe latter phases. Extensive experimental evaluations reveal that TVLARSconsistently outperforms LARS and LAMB in most cases, with improvements of upto 2% in classification scenarios. Notably, in every case of self-supervisedlearning, TVLARS dominates LARS and LAMB with performance improvements of up to10%.

Revisiting LARS for Large Batch Training Generalization of Neural Networks