English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we explore theoretical properties of training a two-layeredReLU network $g(\mathbf{x}; \mathbf{w}) = \sum_{j=1}^K\sigma(\mathbf{w}_j^T\mathbf{x})$ with centered $d$-dimensional sphericalGaussian input $\mathbf{x}$ ($\sigma$=ReLU). We train our network with gradientdescent on $\mathbf{w}$ to mimic the output of a teacher network with the samearchitecture and fixed parameters $\mathbf{w}^*$. We show that its populationgradient has an analytical formula, leading to interesting theoretical analysisof critical points and convergence behaviors. First, we prove that criticalpoints outside the hyperplane spanned by the teacher parameters("out-of-plane") are not isolated and form manifolds, and characterize in-planecritical-point-free regions for two ReLU case. On the other hand, convergenceto $\mathbf{w}^*$ for one ReLU node is guaranteed with at least$(1-\epsilon)/2$ probability, if weights are initialized randomly with standarddeviation upper-bounded by $O(\epsilon/\sqrt{d})$, consistent with empiricalpractice. For network with many ReLU nodes, we prove that an infinitesimalperturbation of weight initialization results in convergence towards$\mathbf{w}^*$ (or its permutation), a phenomenon known as spontaneoussymmetric-breaking (SSB) in physics. We assume no independence of ReLUactivations. Simulation verifies our findings.

An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis