English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Naive backpropagation through time has a memory footprint that grows linearlyin the sequence length, due to the need to store each state of the forwardpropagation. This is a problem for large networks. Strategies have beendeveloped to trade memory for added computations, which results in a sublineargrowth of memory footprint or computation overhead. In this work, we present alibrary that uses asynchronous storing and prefetching to move data to and fromslow and cheap stor- age. The library only stores and prefetches states asfrequently as possible without delaying the computation, and uses the optimalRevolve backpropagation strategy for the computations in between. The memoryfootprint of the backpropagation can thus be reduced to any size (e.g. to fitinto DRAM), while the computational overhead is constant in the sequencelength, and only depends on the ratio between compute and transfer times on agiven hardware. We show in experiments that by exploiting asyncronous datatransfer, our strategy is always at least as fast, and usually faster than thepreviously studied "optimal" strategies.

Backpropagation for long sequences: beyond memory constraints with constant overheads