English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we study the cross-modal image retrieval, where the inputscontain a source image plus some text that describes certain modifications tothis image and the desired image. Prior work usually uses a three-stagestrategy to tackle this task: 1) extract the features of the inputs; 2) fusethe feature of the source image and its modified text to obtain fusion feature;3) learn a similarity metric between the desired image and the source image +modified text by using deep metric learning. Since classical image/textencoders can learn the useful representation and common pair-based lossfunctions of distance metric learning are enough for cross-modal retrieval,people usually improve retrieval accuracy by designing new fusion networks.However, these methods do not successfully handle the modality gap caused bythe inconsistent distribution and representation of the features of differentmodalities, which greatly influences the feature fusion and similaritylearning. To alleviate this problem, we adopt the contrastive self-supervisedlearning method Deep InforMax (DIM) to our approach to bridge this gap byenhancing the dependence between the text, the image, and their fusion.Specifically, our method narrows the modality gap between the text modality andthe image modality by maximizing mutual information between their not exactlysemantically identical representation. Moreover, we seek an effective commonsubspace for the semantically same fusion feature and desired image's featureby utilizing Deep InforMax between the low-level layer of the image encoder andthe high-level layer of the fusion network. Extensive experiments on threelarge-scale benchmark datasets show that we have bridged the modality gapbetween different modalities and achieve state-of-the-art retrievalperformance.

Cross-modal Image Retrieval with Deep Mutual Information Maximization