English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Pretraining auto-regressive large language models (LLMs) with retrievaldemonstrates better perplexity and factual accuracy by leveraging externaldatabases. However, the size of existing pretrained retrieval-augmented LLM isstill limited (e.g., Retro has 7.5B parameters), which limits the effectivenessof instruction tuning and zero-shot generalization. In this work, we introduceRetro 48B, the largest LLM pretrained with retrieval before instruction tuning.Specifically, we continue to pretrain the 43B GPT model on additional 100billion tokens using the Retro augmentation method by retrieving from 1.2trillion tokens. The obtained foundation model, Retro 48B, largely outperformsthe original 43B GPT in terms of perplexity. After instruction tuning on Retro,InstructRetro demonstrates significant improvement over the instruction tunedGPT on zero-shot question answering (QA) tasks. Specifically, the averageimprovement of InstructRetro is 7% over its GPT counterpart across 8 short-formQA tasks, and 10% over GPT across 4 challenging long-form QA tasks.Surprisingly, we find that one can ablate the encoder from InstructRetroarchitecture and directly use its decoder backbone, while achieving comparableresults. We hypothesize that pretraining with retrieval makes its decoder goodat incorporating context for QA. Our results highlights the promising directionto obtain a better GPT decoder for QA through continued pretraining withretrieval before instruction tuning.

InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining