English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

MEGA is a recent transformer-based architecture, which utilizes a linearrecurrent operator whose parallel computation, based on the FFT, scales as$O(LlogL)$, with $L$ being the sequence length. We build upon their approach byreplacing the linear recurrence with a special temporal convolutional networkwhich permits larger receptive field size with shallower networks, and reducesthe computational complexity to $O(L)$. The resulting model is called TCNCA, aTemporal Convolutional Network with Chunked Attention. We evaluate TCNCA onEnWik8 language modeling, long-range-arena (LRA) sequence classification, aswell as a synthetic reasoning benchmark associative recall. On EnWik8, TCNCAoutperforms MEGA, reaching a lower loss with $1.37\times$/$1.24\times$ fasterforward/backward pass during training. The dilated convolutions used in TCNCAare consistently and significantly faster operations than the FFT-basedparallelized recurrence in GPUs, making them a scalable candidate for handlingvery large sequence lengths: they are up to $7.07\times$/$2.86\times$ faster inthe forward/backward pass for sequences up to 131k. Further on LRA, TCNCAachieves, on average, $1.28\times$ speed-up during inference with similaraccuracy to what MEGA achieves. On associative recall, we find that even asimplified version of TCNCA, without excessive multiplicative and additiveinteractions, remains superior or competitive to MEGA on a range of sequencelengths and vocabulary sizes.

TCNCA: Temporal Convolution Network with Chunked Attention for Scalable Sequence Processing