English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large language models (LLMs) are in need of sufficient contexts to handlemany critical applications, such as retrieval augmented generation and few-shotlearning. However, due to the constrained window size, the LLMs can only accessto the information within a limited context. Although the size of contextwindow can be extended by fine-tuning, it will result in a substantial cost inboth training and inference stage. In this paper, we present ExtensibleTokenization as an alternative method which realizes the flexible scaling ofLLMs' context. Extensible Tokenization stands as a midware in between of thetokenized context and the LLM, which transforms the raw token embeddings intothe extensible embeddings. Such embeddings provide a more compactrepresentation for the long context, on top of which the LLM is able toperceive more information with the same context window. Extensible Tokenizationis also featured by its flexibility: the scaling factor can be flexiblydetermined within a feasible scope, leading to the extension of an arbitrarycontext length at the inference time. Besides, Extensible Tokenization isintroduced as a drop-in component, which can be seamlessly plugged into notonly the LLM itself and but also its fine-tuned derivatives, bringing in theextended contextual information while fully preserving the LLM's existingcapabilities. We perform comprehensive experiments on long-context languagemodeling and understanding tasks, which verify Extensible Tokenization as aneffective, efficient, flexible, and compatible method to extend LLM's context.Our model and source code will be made publicly available.

Flexibly Scaling Large Language Models Contexts Through Extensible Tokenization