English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In-Context Learning (ICL) is an emergent capability of Large Language Models(LLMs). Only a few demonstrations enable LLMs to be used as blackbox for newtasks. Previous studies have shown that using LLMs' outputs as labels iseffective in training models to select demonstrations. Such a label is expectedto estimate utility of a demonstration in ICL; however, it has not been wellunderstood how different labeling strategies affect results on target tasks.This paper presents an analysis on different utility functions by focusing onLLMs' output probability given ground-truth output, and task-specific rewardgiven LLMs' prediction. Unlike the previous work, we introduce a novel labelingmethod, incremental utility, which estimates how much incremental knowledge isbrought into the LLMs by a demonstration. We conduct experiments withinstruction-tuned LLMs on binary/multi-class classification, segmentation, andtranslation across Arabic, English, Finnish, Japanese, and Spanish. Our resultsshow that (1) the probability is effective when the probability values aredistributed across the whole value range (on the classification tasks), and (2)the downstream metric is more robust when nuanced reward values are providedwith long outputs (on the segmentation and translation tasks). We then showthat the proposed incremental utility further helps ICL by contrasting how theLLMs perform with and without the demonstrations.

Take One Step at a Time to Know Incremental Utility of Demonstration: An Analysis on Reranking for Few-Shot In-Context Learning