English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent works on instruction tuning (IT) have achieved great performance withzero-shot generalizability to unseen tasks. With additional context (e.g., taskdefinition, examples) provided to models for fine-tuning, they achieved muchhigher performance than untuned models. Despite impressive performance gains,what models learn from IT remains understudied. In this work, we analyze howmodels utilize instructions during IT by comparing model training with alteredvs. original instructions. Specifically, we create simplified task definitionsby removing all semantic components and only leaving the output spaceinformation, and delusive examples that contain incorrect input-output mapping.Our experiments show that models trained on simplified task definition ordelusive examples can achieve comparable performance to the ones trained on theoriginal instructions and examples. Furthermore, we introduce a random baselineto perform zeroshot classification tasks, and find it achieves similarperformance (42.6% exact-match) as IT does (43% exact-match) in low resourcesetting, while both methods outperform naive T5 significantly (30% perexact-match). Our analysis provides evidence that the impressive performancegain of current IT models can come from picking up superficial patterns, suchas learning the output format and guessing. Our study highlights the urgentneed for more reliable IT methods and evaluation.

Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning