English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

How initialization and loss function affect the learning of a deep neuralnetwork (DNN), specifically its generalization error, is an important problemin practice. In this work, by exploiting the linearity of DNN training dynamicsin the NTK regime \citep{jacot2018neural,lee2019wide}, we provide an explicitand quantitative answer to this problem. Focusing on regression problem, weprove that, in the NTK regime, for any loss in a general class of functions,the DNN finds the same \emph{global} minima---the one that is nearest to theinitial value in the parameter space, or equivalently, the one that is closestto the initial DNN output in the corresponding reproducing kernel Hilbertspace. Using these optimization problems, we quantify the impact of initialoutput and prove that a random non-zero one increases the generalization error.We further propose an antisymmetrical initialization (ASI) trick thateliminates this type of error and accelerates the training. To understandwhether the above results hold in general, we also perform experiments for DNNsin the non-NTK regime, which demonstrate the effectiveness of our theoreticalresults and the ASI trick in a qualitative sense. Overall, our work serves as abaseline for the further investigation of the impact of initialization and lossfunction on the generalization of DNNs, which can potentially guide and improvethe training of DNNs in practice.

A type of generalization error induced by initialization in deep neural networks