English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Motivated by the resurgence of neural networks in being able to solve complexlearning tasks we undertake a study of high depth networks using ReLU gateswhich implement the function $x \mapsto \max\{0,x\}$. We try to understand therole of depth in such neural networks by showing size lowerbounds against suchnetwork architectures in parameter regimes hitherto unexplored. In particularwe show the following two main results about neural nets computing Booleanfunctions of input dimension $n$,  1. We use the method of random restrictions to show almost linear,$\Omega(\epsilon^{2(1-\delta)}n^{1-\delta})$, lower bound for completely weightunrestricted LTF-of-ReLU circuits to match the Andreev function on at least$\frac{1}{2} +\epsilon$ fraction of the inputs for $\epsilon >\sqrt{2\frac{\log^{\frac {2}{2-\delta}}(n)}{n}}$ for any $\delta \in (0,\frac 12)$  2. We use the method of sign-rank to show exponential in dimension lowerbounds for ReLU circuits ending in a LTF gate and of depths upto $O(n^{\xi})$with $\xi < \frac{1}{8}$ with some restrictions on the weights in the bottommost layer. All other weights in these circuits are kept unrestricted. This inturns also implies the same lowerbounds for LTF circuits with the samearchitecture and the same weight restrictions on their bottom most layer.  Along the way we also show that there exists a $\mathbb{R}^ n\rightarrow\mathbb{R}$ Sum-of-ReLU-of-ReLU function which Sum-of-ReLU neural nets cannever represent no matter how large they are allowed to be.

Lower bounds over Boolean inputs for deep neural networks with ReLU gates