English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

The application of zero-shot learning in computer vision has beenrevolutionized by the use of image-text matching models. The most notableexample, CLIP, has been widely used for both zero-shot classification andguiding generative models with a text prompt. However, the zero-shot use ofCLIP is unstable with respect to the phrasing of the input text, making itnecessary to carefully engineer the prompts used. We find that this instabilitystems from a selective similarity score, which is based only on a subset of thesemantically meaningful input tokens. To mitigate it, we present a novelexplainability-based approach, which adds a loss term to ensure that CLIPfocuses on all relevant semantic parts of the input, in addition to employingthe CLIP similarity loss used in previous works. When applied to one-shotclassification through prompt engineering, our method yields an improvement inthe recognition rate, without additional training or fine-tuning. Additionally,we show that CLIP guidance of generative models using our method significantlyimproves the generated images. Finally, we demonstrate a novel use of CLIPguidance for text-based image generation with spatial conditioning on objectlocation, by requiring the image explainability heatmap for each object to beconfined to a pre-determined bounding box.

No Token Left Behind: Explainability-Aided Image Classification and Generation