English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Scaling Transformers to longer sequence lengths has been a major problem inthe last several years, promising to improve performance in language modelingand high-resolution image understanding, as well as to unlock new applicationsin code, audio, and video generation. The attention layer is the mainbottleneck in scaling to longer sequences, as its runtime and memory increasequadratically in the sequence length. FlashAttention exploits the asymmetricGPU memory hierarchy to bring significant memory saving (linear instead ofquadratic) and runtime speedup (2-4$\times$ compared to optimized baselines),with no approximation. However, FlashAttention is still not nearly as fast asoptimized matrix-multiply (GEMM) operations, reaching only 25-40\% of thetheoretical maximum FLOPs/s. We observe that the inefficiency is due tosuboptimal work partitioning between different thread blocks and warps on theGPU, causing either low-occupancy or unnecessary shared memory reads/writes. Wepropose FlashAttention-2, with better work partitioning to address theseissues. In particular, we (1) tweak the algorithm to reduce the number ofnon-matmul FLOPs (2) parallelize the attention computation, even for a singlehead, across different thread blocks to increase occupancy, and (3) within eachthread block, distribute the work between warps to reduce communication throughshared memory. These yield around 2$\times$ speedup compared to FlashAttention,reaching 50-73\% of the theoretical maximum FLOPs/s on A100 and getting closeto the efficiency of GEMM operations. We empirically validate that when usedend-to-end to train GPT-style models, FlashAttention-2 reaches training speedof up to 225 TFLOPs/s per A100 GPU (72\% model FLOPs utilization).

FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning