English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Contrastive learning-based vision-language pre-training approaches, such asCLIP, have demonstrated great success in many vision-language tasks. Thesemethods achieve cross-modal alignment by encoding a matched image-text pairwith similar feature embeddings, which are generated by aggregating informationfrom visual patches and language tokens. However, direct aligning cross-modalinformation using such representations is challenging, as visual patches andtext tokens differ in semantic levels and granularities. To alleviate thisissue, we propose a Finite Discrete Tokens (FDT) based multimodalrepresentation. FDT is a set of learnable tokens representing certainvisual-semantic concepts. Both images and texts are embedded using shared FDTby first grounding multimodal inputs to FDT space and then aggregating theactivated FDT representations. The matched visual and semantic concepts areenforced to be represented by the same set of discrete tokens by a sparseactivation constraint. As a result, the granularity gap between the twomodalities is reduced. Through both quantitative and qualitative analyses, wedemonstrate that using FDT representations in CLIP-style models improvescross-modal alignment and performance in visual recognition and vision-languagedownstream tasks. Furthermore, we show that our method can learn morecomprehensive representations, and the learned FDT capture meaningfulcross-modal correspondence, ranging from objects to actions and attributes.

Revisiting Multimodal Representation in Contrastive Learning: From Patch and Token Embeddings to Finite Discrete Tokens