English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In recent years, vision and language pre-training (VLP) models have advancedthe state-of-the-art results in a variety of cross-modal downstream tasks.Aligning cross-modal semantics is claimed to be one of the essentialcapabilities of VLP models. However, it still remains unclear about the innerworking mechanism of alignment in VLP models. In this paper, we propose a newprobing method that is based on image captioning to first empirically study thecross-modal semantics alignment of VLP models. Our probing method is built uponthe fact that given an image-caption pair, the VLP models will give a score,indicating how well two modalities are aligned; maximizing such scores willgenerate sentences that VLP models believe are of good alignment. Analyzingthese sentences thus will reveal in what way different modalities are alignedand how well these alignments are in VLP models. We apply our probing method tofive popular VLP models, including UNITER, ROSITA, ViLBERT, CLIP, and LXMERT,and provide a comprehensive analysis of the generated captions guided by thesemodels. Our results show that VLP models (1) focus more on just aligningobjects with visual words, while neglecting global semantics; (2) prefer fixedsentence patterns, thus ignoring more important textual information includingfluency and grammar; and (3) deem the captions with more visual words arebetter aligned with images. These findings indicate that VLP models still haveweaknesses in cross-modal semantics alignment and we hope this work will drawresearchers' attention to such problems when designing a new VLP model.

Probing Cross-modal Semantics Alignment Capability from the Textual Perspective