English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We show that the empirical risk minimization (ERM) problem for neuralnetworks has no solution in general. Given a training set $s_1, \dots, s_n \in\mathbb{R}^p$ with corresponding responses $t_1,\dots,t_n \in \mathbb{R}^q$,fitting a $k$-layer neural network $\nu_\theta : \mathbb{R}^p \to \mathbb{R}^q$involves estimation of the weights $\theta \in \mathbb{R}^m$ via an ERM: \[\inf_{\theta \in \mathbb{R}^m} \; \sum_{i=1}^n \lVert t_i - \nu_\theta(s_i)\rVert_2^2. \] We show that even for $k = 2$, this infimum is not attainable ingeneral for common activations like ReLU, hyperbolic tangent, and sigmoidfunctions. A high-level explanation is like that for the nonexistence of bestrank-$r$ approximations of higher-order tensors --- the set of parameters isnot a closed set --- but the geometry involved for best $k$-layer neuralnetworks approximations is more subtle. In addition, we show that for smoothactivations $\sigma(x)= 1/\bigl(1 + \exp(-x)\bigr)$ and $\sigma(x)=\tanh(x)$,such failure to attain an infimum can happen on a positive-measured subset ofresponses. For the ReLU activation $\sigma(x)=\max(0,x)$, we completelyclassifying cases where the ERM for a best two-layer neural networkapproximation attains its infimum. As an aside, we obtain a precise descriptionof the geometry of the space of two-layer neural networks with $d$ neurons inthe hidden layer: it is the join locus of a line and the $d$-secant locus of acone.

Best k-layer neural network approximations