English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Aggregating multi-modality data to obtain reliable data representationattracts more and more attention. Recent studies demonstrate that Transformermodels usually work well for multi-modality tasks. Existing Transformersgenerally either adopt the Cross-Attention (CA) mechanism or simpleconcatenation to achieve the information interaction among different modalitieswhich generally ignore the issue of modality gap. In this work, we re-thinkTransformer and extend it to MutualFormer for multi-modality datarepresentation. Rather than CA in Transformer, MutualFormer employs our newdesign of Cross-Diffusion Attention (CDA) to conduct the informationcommunication among different modalities. Comparing with CA, the mainadvantages of the proposed CDA are three aspects. First, the crossaffinities inCDA are defined based on the individual modality affinities in the metric spacewhich thus can naturally avoid the issue of modality/domain gap in featurebased CA definition. Second, CDA provides a general scheme which can either beused for multimodality representation or serve as the post-optimization forexisting CA models. Third, CDA is implemented efficiently. We successfullyapply the MutualFormer on different multi-modality learning tasks (i.e.,RGB-Depth SOD, RGB-NIR object ReID). Extensive experiments demonstrate theeffectiveness of the proposed MutualFormer.

MutualFormer: Multi-Modality Representation Learning via Cross-Diffusion Attention