English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Contrastive vision-language representation learning has achievedstate-of-the-art performance for zero-shot classification, by learning frommillions of image-caption pairs crawled from the internet. However, the massivedata that powers large multimodal models such as CLIP, makes them extremelyvulnerable to various types of targeted data poisoning and backdoor attacks.Despite this vulnerability, robust contrastive vision-language pre-trainingagainst such attacks has remained unaddressed. In this work, we propose ROCLIP,the first effective method for robust pre-training multimodal vision-languagemodels against targeted data poisoning and backdoor attacks. ROCLIP effectivelybreaks the association between poisoned image-caption pairs by considering arelatively large and varying pool of random captions, and matching every imagewith the text that is most similar to it in the pool instead of its owncaption, every few epochs.It also leverages image and text augmentations tofurther strengthen the defense and improve the performance of the model. Ourextensive experiments show that ROCLIP renders state-of-the-art targeted datapoisoning and backdoor attacks ineffective during pre-training CLIP models. Inparticular, ROCLIP decreases the success rate for targeted data poisoningattacks from 93.75% to 12.5% and that of backdoor attacks down to 0%, whileimproving the model's linear probe performance by 10% and maintains a similarzero shot performance compared to CLIP. By increasing the frequency ofmatching, ROCLIP is able to defend strong attacks, which add up to 1% poisonedexamples to the data, and successfully maintain a low attack success rate of12.5%, while trading off the performance on some tasks.

Robust Contrastive Language-Image Pre-training against Data Poisoning and Backdoor Attacks