English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recurrent neural networks (RNNs) are well suited for solving sequence tasksin resource-constrained systems due to their expressivity and low computationalrequirements. However, there is still a need to bridge the gap between whatRNNs are capable of in terms of efficiency and performance and real-worldapplication requirements. The memory and computational requirements arisingfrom propagating the activations of all the neurons at every time step to everyconnected neuron, together with the sequential dependence of activations,contribute to the inefficiency of training and using RNNs. We propose asolution inspired by biological neuron dynamics that makes the communicationbetween RNN units sparse and discrete. This makes the backward pass withbackpropagation through time (BPTT) computationally sparse and efficient aswell. We base our model on the gated recurrent unit (GRU), extending it withunits that emit discrete events for communication triggered by a threshold sothat no information is communicated to other units in the absence of events. Weshow theoretically that the communication between units, and hence thecomputation required for both the forward and backward passes, scales with thenumber of events in the network. Our model achieves efficiency withoutcompromising task performance, demonstrating competitive performance comparedto state-of-the-art recurrent network models in real-world tasks, includinglanguage modeling. The dynamic activity sparsity mechanism also makes our modelwell suited for novel energy-efficient neuromorphic hardware. Code is availableat https://github.com/KhaleelKhan/EvNN/.

Efficient recurrent architectures through activity sparsity and sparse back-propagation through time