English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

A recent line of work shows that a deep neural network with ReLUnonlinearities arises from a finite sequence of cascaded sparse coding models,the outputs of which, except for the last element in the cascade, are sparseand unobservable. That is, intermediate outputs deep in the cascade are sparse,hence the title of this manuscript. We show here, using techniques from thedictionary learning literature that, if the measurement matrices in thecascaded sparse coding model (a) satisfy RIP and (b) all have sparse columnsexcept for the last, they can be recovered with high probability. We proposetwo algorithms for this purpose: one that recovers the matrices in a forwardsequence, and another that recovers them in a backward sequence. The method ofchoice in deep learning to solve this problem is by training an auto-encoder.Our algorithms provide a sound alternative, with theoretical guarantees, aswell upper bounds on sample complexity. The theory shows that the learningcomplexity of the forward algorithm depends on the number of hidden units atthe deepest layer and the number of active neurons at that layer (sparsity). Inaddition, the theory relates the number of hidden units in successive layers,thus giving a practical prescription for designing deep ReLU neural networks.Because it puts fewer restrictions on the architecture, the backward algorithmrequires more data. We demonstrate the deep dictionary learning algorithm viasimulations. Finally, we use a coupon-collection argument to conjecture a lowerbound on sample complexity that gives some insight as to why deep networksrequire more data to train than shallow ones.

Deeply-Sparse Signal rePresentations ($\text{D}\text{S}^2\text{P}$)