English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We present a Multimodal Backdoor Defense technique TIJO (Trigger Inversionusing Joint Optimization). Recent work arXiv:2112.07668 has demonstratedsuccessful backdoor attacks on multimodal models for the Visual QuestionAnswering task. Their dual-key backdoor trigger is split across two modalities(image and text), such that the backdoor is activated if and only if thetrigger is present in both modalities. We propose TIJO that defends againstdual-key attacks through a joint optimization that reverse-engineers thetrigger in both the image and text modalities. This joint optimization ischallenging in multimodal models due to the disconnected nature of the visualpipeline which consists of an offline feature extractor, whose output is thenfused with the text using a fusion module. The key insight enabling the jointoptimization in TIJO is that the trigger inversion needs to be carried out inthe object detection box feature space as opposed to the pixel space. Wedemonstrate the effectiveness of our method on the TrojVQA benchmark, whereTIJO improves upon the state-of-the-art unimodal methods from an AUC of 0.6 to0.92 on multimodal dual-key backdoors. Furthermore, our method also improvesupon the unimodal baselines on unimodal backdoors. We present ablation studiesand qualitative results to provide insights into our algorithm such as thecritical importance of overlaying the inverted feature triggers on all visualfeatures during trigger inversion. The prototype implementation of TIJO isavailable at https://github.com/SRI-CSL/TIJO.

TIJO: Trigger Inversion with Joint Optimization for Defending Multimodal Backdoored Models