English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

The fusion of vision and language has brought about a transformative shift incomputer vision through the emergence of Vision-Language Models (VLMs).However, the resource-intensive nature of existing VLMs poses a significantchallenge. We need an accessible method for developing the next generation ofVLMs. To address this issue, we propose Zoom-shot, a novel method fortransferring the zero-shot capabilities of CLIP to any pre-trained visionencoder. We do this by exploiting the multimodal information (i.e. text andimage) present in the CLIP latent space through the use of specificallydesigned multimodal loss functions. These loss functions are (1)cycle-consistency loss and (2) our novel prompt-guided knowledge distillationloss (PG-KD). PG-KD combines the concept of knowledge distillation with CLIP'szero-shot classification, to capture the interactions between text and imagefeatures. With our multimodal losses, we train a $\textbf{linear mapping}$between the CLIP latent space and the latent space of a pre-trained visionencoder, for only a $\textbf{single epoch}$. Furthermore, Zoom-shot is entirelyunsupervised and is trained using $\textbf{unpaired}$ data. We test thezero-shot capabilities of a range of vision encoders augmented as new VLMs, oncoarse and fine-grained classification datasets, outperforming the previousstate-of-the-art in this problem domain. In our ablations, we find Zoom-shotallows for a trade-off between data and compute during training; and ourstate-of-the-art results can be obtained by reducing training from 20% to 1% ofthe ImageNet training data with 20 epochs. All code and models are available onGitHub.

Zoom-shot: Fast and Efficient Unsupervised Zero-Shot Transfer of CLIP to Vision Encoders with Multimodal Loss