English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Self-attention and position embedding are two key modules intransformer-based Large Language Models (LLMs). However, the potentialrelationship between them is far from well studied, especially for long contextwindow extending. In fact, anomalous behaviors harming long contextextrapolation exist between Rotary Position Embedding (RoPE) and vanillaself-attention unveiled by our work. To address this issue, we propose a novelattention mechanism, CoCA (Collinear Constrained Attention). Specifically, weenforce a collinear constraint between $Q$ and $K$ to seamlessly integrate RoPEand self-attention. While only adding minimal computational and spatialcomplexity, this integration significantly enhances long context windowextrapolation ability. We provide an optimized implementation, making it adrop-in replacement for any existing transformer-based models. Extensiveexperiments show that CoCA performs extraordinarily well in extending contextwindows. A CoCA-based GPT model, trained with a context length of 512, canseamlessly extend the context window up to 32K (60$\times$), without anyfine-tuning. Additionally, by dropping CoCA in LLaMA-7B, we achieveextrapolation up to 32K within only 2K training length. Our code is publiclyavailable at: https://github.com/codefuse-ai/Collinear-Constrained-Attention

CoCA: Fusing Position Embedding with Collinear Constrained Attention in Transformers for Long Context Window Extending