English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The expressivity of neural networks as a function of their depth, width andtype of activation units has been an important question in deep learningtheory. Recently, depth separation results for ReLU networks were obtained viaa new connection with dynamical systems, using a generalized notion of fixedpoints of a continuous map $f$, called periodic points. In this work, westrengthen the connection with dynamical systems and we improve the existingwidth lower bounds along several aspects. Our first main result isperiod-specific width lower bounds that hold under the stronger notion of$L^1$-approximation error, instead of the weaker classification error. Oursecond contribution is that we provide sharper width lower bounds, stillyielding meaningful exponential depth-width separations, in regimes whereprevious results wouldn't apply. A byproduct of our results is that thereexists a universal constant characterizing the depth-width trade-offs, as longas $f$ has odd periods. Technically, our results follow by unveiling a tighterconnection between the following three quantities of a given function: itsperiod, its Lipschitz constant and the growth rate of the number ofoscillations arising under compositions of the function $f$ with itself.

Better Depth-Width Trade-offs for Neural Networks through the lens of Dynamical Systems