English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Pretrained language models have achieved state-of-the-art performance whenadapted to a downstream NLP task. However, theoretical analysis of these modelsis scarce and challenging since the pretraining and downstream tasks can bevery different. We propose an analysis framework that links the pretraining anddownstream tasks with an underlying latent variable generative model of text --the downstream classifier must recover a function of the posterior distributionover the latent variables. We analyze head tuning (learning a classifier on topof the frozen pretrained model) and prompt tuning in this setting. Thegenerative model in our analysis is either a Hidden Markov Model (HMM) or anHMM augmented with a latent memory component, motivated by long-termdependencies in natural language. We show that 1) under certain non-degeneracyconditions on the HMM, simple classification heads can solve the downstreamtask, 2) prompt tuning obtains downstream guarantees with weaker non-degeneracyconditions, and 3) our recovery guarantees for the memory-augmented HMM arestronger than for the vanilla HMM because task-relevant information is easierto recover from the long-term memory. Experiments on synthetically generateddata from HMMs back our theoretical findings.

Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning