English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The popular CLIP model displays impressive zero-shot capabilities thanks toits seamless interaction with arbitrary text prompts. However, its lack ofspatial awareness makes it unsuitable for dense computer vision tasks, e.g.,semantic segmentation, without an additional fine-tuning step that often usesannotations and can potentially suppress its original open-vocabularyproperties. Meanwhile, self-supervised representation methods have demonstratedgood localization properties without human-made annotations nor explicitsupervision. In this work, we take the best of both worlds and propose azero-shot open-vocabulary semantic segmentation method, which does not requireany annotations. We propose to locally improve dense MaskCLIP features,computed with a simple modification of CLIP's last pooling layer, byintegrating localization priors extracted from self-supervised features. Bydoing so, we greatly improve the performance of MaskCLIP and produce smoothoutputs. Moreover, we show that the used self-supervised feature properties candirectly be learnt from CLIP features therefore allowing us to obtain the bestresults with a single pass through CLIP model. Our method CLIP-DINOiser needsonly a single forward pass of CLIP and two light convolutional layers atinference, no extra supervision nor extra memory and reaches state-of-the-artresults on challenging and fine-grained benchmarks such as COCO, PascalContext, Cityscapes and ADE20k. The code to reproduce our results is availableat https://github.com/wysoczanska/clip_dinoiser.

CLIP-DINOiser: Teaching CLIP a few DINO tricks