English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Quasi-Newton methods still face significant challenges in traininglarge-scale neural networks due to additional compute costs in the Hessianrelated computations and instability issues in stochastic training. Awell-known method, L-BFGS that efficiently approximates the Hessian usinghistory parameter and gradient changes, suffers convergence instability instochastic training. So far, attempts that adapt L-BFGS to large-scalestochastic training incur considerable extra overhead, which offsets itsconvergence benefits in wall-clock time. In this paper, we propose mL-BFGS, alightweight momentum-based L-BFGS algorithm that paves the way for quasi-Newton(QN) methods in large-scale distributed deep neural network (DNN) optimization.mL-BFGS introduces a nearly cost-free momentum scheme into L-BFGS update andgreatly reduces stochastic noise in the Hessian, therefore stabilizingconvergence during stochastic optimization. For model training at a largescale, mL-BFGS approximates a block-wise Hessian, thus enabling distributingcompute and memory costs across all computing nodes. We provide a supportingconvergence analysis for mL-BFGS in stochastic settings. To investigate mL-BFGSpotential in large-scale DNN training, we train benchmark neural models usingmL-BFGS and compare performance with baselines (SGD, Adam, and otherquasi-Newton methods). Results show that mL-BFGS achieves both noticeableiteration-wise and wall-clock speedup.

mL-BFGS: A Momentum-based L-BFGS for Distributed Large-Scale Neural Network Optimization