English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Large-batch stochastic gradient descent (SGD) is widely used for training indistributed deep learning because of its training-time efficiency, however,extremely large-batch SGD leads to poor generalization and easily converges tosharp minima, which prevents naive large-scale data-parallel SGD (DP-SGD) fromconverging to good minima. To overcome this difficulty, we propose gradientnoise convolution (GNC), which effectively smooths sharper minima of the lossfunction. For DP-SGD, GNC utilizes so-called gradient noise, which is inducedby stochastic gradient variation and convolved to the loss function as asmoothing effect. GNC computation can be performed by simply computing thestochastic gradient on each parallel worker and merging them, and is thereforeextremely easy to implement. Due to convolving with the gradient noise, whichtends to spread along a sharper direction of the loss function, GNC caneffectively smooth sharp minima and achieve better generalization, whereasisotropic random noise cannot. We empirically show this effect by comparing GNCwith isotropic random noise, and show that it achieves state-of-the-artgeneralization performance for large-scale deep neural network optimization.

Gradient Noise Convolution (GNC): Smoothing Loss Function for Distributed Large-Batch SGD