English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Cross-modal retrieval has become popular in recent years, particularly withthe rise of multimedia. Generally, the information from each modality exhibitsdistinct representations and semantic information, which makes feature tends tobe in separate latent spaces encoded with dual-tower architecture and makes itdifficult to establish semantic relationships between modalities, resulting inpoor retrieval performance. To address this issue, we propose a novel frameworkfor cross-modal retrieval which consists of a cross-modal mixer, a maskedautoencoder for pre-training, and a cross-modal retriever for downstreamtasks.In specific, we first adopt cross-modal mixer and mask modeling to fusethe original modality and eliminate redundancy. Then, an encoder-decoderarchitecture is applied to achieve a fuse-then-separate task in thepre-training phase.We feed masked fused representations into the encoder andreconstruct them with the decoder, ultimately separating the original data oftwo modalities. In downstream tasks, we use the pre-trained encoder to buildthe cross-modal retrieval method. Extensive experiments on 2 real-worlddatasets show that our approach outperforms previous state-of-the-art methodsin video-audio matching tasks, improving retrieval accuracy by up to 2 times.Furthermore, we prove our model performance by transferring it to otherdownstream tasks as a universal model.

Video and Audio are Images: A Cross-Modal Mixer for Original Data on Video-Audio Retrieval