English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

With multiple iterations of updates, local statistical gradient descent(L-SGD) has been proven to be very effective in distributed machine learningschemes such as federated learning. In fact, many innovative works have shownthat L-SGD with independent and identically distributed (IID) data can evenoutperform SGD. As a result, extensive efforts have been made to unveil thepower of L-SGD. However, existing analysis failed to explain why the multiplelocal updates with small mini-batches of data (L-SGD) can not be replaced bythe update with one big batch of data and a larger learning rate (SGD). In thispaper, we offer a new perspective to understand the strength of L-SGD. Wetheoretically prove that, with IID data, L-SGD can effectively explore thesecond order information of the loss function. In particular, compared withSGD, the updates of L-SGD have much larger projection on the eigenvectors ofthe Hessian matrix with small eigenvalues, which leads to faster convergence.Under certain conditions, L-SGD can even approach the Newton method. Experimentresults over two popular datasets validate the theoretical results.

Local SGD Accelerates Convergence by Exploiting Second Order Information of the Loss Function