English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The ability to quickly learn a new task with minimal instruction - known asfew-shot learning - is a central aspect of intelligent agents. Classicalfew-shot benchmarks make use of few-shot samples from a single modality, butsuch samples may not be sufficient to characterize an entire concept class. Incontrast, humans use cross-modal information to learn new concepts efficiently.In this work, we demonstrate that one can indeed build a better ${\bf visual}$dog classifier by ${\bf read}$ing about dogs and ${\bf listen}$ing to thembark. To do so, we exploit the fact that recent multimodal foundation modelssuch as CLIP are inherently cross-modal, mapping different modalities to thesame representation space. Specifically, we propose a simple cross-modaladaptation approach that learns from few-shot examples spanning differentmodalities. By repurposing class names as additional one-shot training samples,we achieve SOTA results with an embarrassingly simple linear classifier forvision-language adaptation. Furthermore, we show that our approach can benefitexisting methods such as prefix tuning, adapters, and classifier ensembling.Finally, to explore other modalities beyond vision and language, we constructthe first (to our knowledge) audiovisual few-shot benchmark and use cross-modaltraining to improve the performance of both image and audio classification.

Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models