English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Despite perfectly interpolating the training data, deep neural networks(DNNs) can often generalize fairly well, in part due to the "implicitregularization" induced by the learning algorithm. Nonetheless, various formsof regularization, such as "explicit regularization" (via weight decay), areoften used to avoid overfitting, especially when the data is corrupted. Thereare several challenges with explicit regularization, most notably unclearconvergence properties. Inspired by convergence properties of stochastic mirrordescent (SMD) algorithms, we propose a new method for training DNNs withregularization, called regularizer mirror descent (RMD). In highlyoverparameterized DNNs, SMD simultaneously interpolates the training data andminimizes a certain potential function of the weights. RMD starts with astandard cost which is the sum of the training loss and a convex regularizer ofthe weights. Reinterpreting this cost as the potential of an "augmented"overparameterized network and applying SMD yields RMD. As a result, RMDinherits the properties of SMD and provably converges to a point "close" to theminimizer of this cost. RMD is computationally comparable to stochasticgradient descent (SGD) and weight decay, and is parallelizable in the samemanner. Our experimental results on training sets with various levels ofcorruption suggest that the generalization performance of RMD is remarkablyrobust and significantly better than both SGD and weight decay, whichimplicitly and explicitly regularize the $\ell_2$ norm of the weights. RMD canalso be used to regularize the weights to a desired weight vector, which isparticularly relevant for continual learning.

Explicit Regularization via Regularizer Mirror Descent