English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Modern supervised learning techniques, particularly those using deep nets,involve fitting high dimensional labelled data sets with functions containingvery large numbers of parameters. Much of this work is empirical. Interestingphenomena have been observed that require theoretical explanations; however thenon-convexity of the loss functions complicates the analysis. Recently it hasbeen proposed that the success of these techniques rests partly in theeffectiveness of the simple stochastic gradient descent algorithm in the socalled interpolation limit in which all labels are fit perfectly. This analysisis made possible since the SGD algorithm reduces to a stochastic linear systemnear the interpolating minimum of the loss function. Here we exploit thisinsight by presenting and analyzing a new distributed algorithm for gradientdescent, also in the interpolating limit. The distributed SGD algorithmpresented in the paper corresponds to gradient descent applied to a simplepenalized distributed loss function, $L({\bf w}_1,...,{\bf w}_n) = \Sigma_il_i({\bf w}_i) + \mu \sum_{<i,j>}|{\bf w}_i-{\bf w}_j|^2$. Here each node holdsonly one sample, and its own parameter vector. The notation $<i,j>$ denotesedges of a connected graph defining the links between nodes. It is shown thatthis distributed algorithm converges linearly (ie the error reducesexponentially with iteration number), with a rate$1-\frac{\eta}{n}\lambda_{min}(H)<R<1$ where $\lambda_{min}(H)$ is the smallestnonzero eigenvalue of the sample covariance or the Hessian H. In contrast withprevious usage of similar penalty functions to enforce consensus between nodes,in the interpolating limit it is not required to take the penalty parameter toinfinity for consensus to occur. The analysis further reinforces the utility ofthe interpolation limit in the theoretical treatment of modern machine learningalgorithms.

Fast Convergence for Stochastic and Distributed Gradient Descent in the Interpolation Limit