English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

This paper explores the complexity of deep feedforward networks with linearpre-synaptic couplings and rectified linear activations. This is a contributionto the growing body of work contrasting the representational power of deep andshallow network architectures. In particular, we offer a framework forcomparing deep and shallow models that belong to the family of piecewise linearfunctions based on computational geometry. We look at a deep rectifiermulti-layer perceptron (MLP) with linear outputs units and compare it with asingle layer version of the model. In the asymptotic regime, when the number ofinputs stays constant, if the shallow model has $kn$ hidden units and $n_0$inputs, then the number of linear regions is $O(k^{n_0}n^{n_0})$. For a $k$layer model with $n$ hidden units on each layer it is $\Omega(\left\lfloor{n}/{n_0}\right\rfloor^{k-1}n^{n_0})$. The number$\left\lfloor{n}/{n_0}\right\rfloor^{k-1}$ grows faster than $k^{n_0}$ when $n$tends to infinity or when $k$ tends to infinity and $n \geq 2n_0$.Additionally, even when $k$ is small, if we restrict $n$ to be $2n_0$, we canshow that a deep model has considerably more linear regions that a shallow one.We consider this as a first step towards understanding the complexity of thesemodels and specifically towards providing suitable mathematical tools forfuture analysis.

On the number of response regions of deep feed forward networks with piece-wise linear activations