English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Neural networks with rectified linear unit activations are essentiallymultivariate linear splines. As such, one of many ways to measure the"complexity" or "expressivity" of a neural network is to count the number ofknots in the spline model. We study the number of knots in fully-connectedfeedforward neural networks with rectified linear unit activation functions. Weintentionally keep the neural networks very simple, so as to make theoreticalanalyses more approachable. An induction on the number of layers $l$ reveals atight upper bound on the number of knots in $\mathbb{R} \to \mathbb{R}^p$ deepneural networks. With $n_i \gg 1$ neurons in layer $i = 1, \dots, l$, the upperbound is approximately $n_1 \dots n_l$. We then show that the exact upper boundis tight, and we demonstrate the upper bound with an example. The purpose ofthese analyses is to pave a path for understanding the behavior of general$\mathbb{R}^q \to \mathbb{R}^p$ neural networks.

The Upper Bound on Knots in Neural Networks