English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Large Language Models (LLMs) are trained with a pre-defined context length,restricting their use in scenarios requiring long inputs. Previous efforts foradapting LLMs to a longer length usually requires fine-tuning with this targetlength (Full-length fine-tuning), suffering intensive training cost. Todecouple train length from target length for efficient context windowextension, we propose Positional Skip-wisE (PoSE) training that smartlysimulates long inputs using a fixed context window. This is achieved by firstdividing the original context window into several chunks, then designingdistinct skipping bias terms to manipulate the position indices of each chunk.These bias terms and the lengths of each chunk are altered for every trainingexample, allowing the model to adapt to all positions within target length.Experimental results show that PoSE greatly reduces memory and time overheadcompared with Full-length fine-tuning, with minimal impact on performance.Leveraging this advantage, we have successfully extended the LLaMA model to128k tokens using a 2k training context window. Furthermore, we empiricallyconfirm that PoSE is compatible with all RoPE-based LLMs and positioninterpolation strategies. Notably, our method can potentially support infinitelength, limited only by memory usage in inference. With ongoing progress forefficient inference, we believe PoSE can further scale the context windowbeyond 128k.

PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training