English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The ability to model intra-modal and inter-modal interactions is fundamentalin multimodal machine learning. The current state-of-the-art models usuallyadopt deep learning models with fixed structures. They can achieve exceptionalperformances on specific tasks, but face a particularly challenging problem ofmodality mismatch because of diversity of input modalities and their fixedstructures. In this paper, we present \textbf{Switch-BERT} for joint vision andlanguage representation learning to address this problem. Switch-BERT extendsBERT architecture by introducing learnable layer-wise and cross-layerinteractions. It learns to optimize attention from a set of attention modesrepresenting these interactions. One specific property of the model is that itlearns to attend outputs from various depths, therefore mitigates the modalitymismatch problem. We present extensive experiments on visual questionanswering, image-text retrieval and referring expression comprehensionexperiments. Results confirm that, whereas alternative architectures includingViLBERT and UNITER may excel in particular tasks, Switch-BERT can consistentlyachieve better or comparable performances than the current state-of-the-artmodels in these tasks. Ablation studies indicate that the proposed modelachieves superior performances due to its ability in learning task-specificmultimodal interactions.

Switch-BERT: Learning to Model Multimodal Interactions by Switching Attention and Input