English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Convolution models with long filters have demonstrated state-of-the-artreasoning abilities in many long-sequence tasks but lag behind the mostoptimized Transformers in wall-clock time. A major bottleneck is the FastFourier Transform (FFT)--which allows long convolutions to run in $O(N logN)$time in sequence length $N$ but has poor hardware utilization. In this paper,we study how to optimize the FFT convolution. We find two key bottlenecks: theFFT does not effectively use specialized matrix multiply units, and it incursexpensive I/O between layers of the memory hierarchy. In response, we proposeFlashFFTConv. FlashFFTConv uses a matrix decomposition that computes the FFTusing matrix multiply units and enables kernel fusion for long sequences,reducing I/O. We also present two sparse convolution algorithms--1) partialconvolutions and 2) frequency-sparse convolutions--which can be implementedsimply by skipping blocks in the matrix decomposition, enabling furtheropportunities for memory and compute savings. FlashFFTConv speeds up exact FFTconvolutions by up to 7.93$\times$ over PyTorch and achieves up to 4.4$\times$speedup end-to-end. Given the same compute budget, FlashFFTConv allowsHyena-GPT-s to achieve 2.3 points better perplexity on the PILE andM2-BERT-base to achieve 3.3 points higher GLUE score--matching models withtwice the parameter count. FlashFFTConv also achieves 96.1% accuracy onPath-512, a high-resolution vision task where no model had previously achievedbetter than 50%. Furthermore, partial convolutions enable longer-sequencemodels--yielding the first DNA model that can process the longest human genes(2.3M base pairs)--and frequency-sparse convolutions speed up pretrained modelswhile maintaining or improving model quality.

FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores