English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In-context learning (ICL) has become the default method for using largelanguage models (LLMs), making the exploration of its limitations andunderstanding the underlying causes crucial. In this paper, we find that ICLfalls short of handling specification-heavy tasks, which are tasks withcomplicated and extensive task specifications, requiring several hours forordinary humans to master, such as traditional information extraction tasks.The performance of ICL on these tasks mostly cannot reach half of thestate-of-the-art results. To explore the reasons behind this failure, weconduct comprehensive experiments on 18 specification-heavy tasks with variousLLMs and identify three primary reasons: inability to specifically understandcontext, misalignment in task schema comprehension with humans, and inadequatelong-text understanding ability. Furthermore, we demonstrate that throughfine-tuning, LLMs can achieve decent performance on these tasks, indicatingthat the failure of ICL is not an inherent flaw of LLMs, but rather a drawbackof existing alignment methods that renders LLMs incapable of handlingcomplicated specification-heavy tasks via ICL. To substantiate this, we performdedicated instruction tuning on LLMs for these tasks and observe a notableimprovement. We hope the analyses in this paper could facilitate advancementsin alignment methods enabling LLMs to meet more sophisticated human demands.

When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks