English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Multimodal contrastive learning aims to train a general-purpose featureextractor, such as CLIP, on vast amounts of raw, unlabeled paired image-textdata. This can greatly benefit various complex downstream tasks, includingcross-modal image-text retrieval and image classification. Despite itspromising prospect, the security issue of cross-modal pre-trained encoder hasnot been fully explored yet, especially when the pre-trained encoder ispublicly available for commercial use.  In this work, we propose AdvCLIP, the first attack framework for generatingdownstream-agnostic adversarial examples based on cross-modal pre-trainedencoders. AdvCLIP aims to construct a universal adversarial patch for a set ofnatural images that can fool all the downstream tasks inheriting the victimcross-modal pre-trained encoder. To address the challenges of heterogeneitybetween different modalities and unknown downstream tasks, we first build atopological graph structure to capture the relevant positions between targetsamples and their neighbors. Then, we design a topology-deviation basedgenerative adversarial network to generate a universal adversarial patch. Byadding the patch to images, we minimize their embeddings similarity todifferent modality and perturb the sample distribution in the feature space,achieving unviersal non-targeted attacks. Our results demonstrate the excellentattack performance of AdvCLIP on two types of downstream tasks across eightdatasets. We also tailor three popular defenses to mitigate AdvCLIP,highlighting the need for new defense mechanisms to defend cross-modalpre-trained encoders.

AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learning