English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Contrastive learning-based video-language representation learning approaches,e.g., CLIP, have achieved outstanding performance, which pursue semanticinteraction upon pre-defined video-text pairs. To clarify this coarse-grainedglobal interaction and move a step further, we have to encounter challengingshell-breaking interactions for fine-grained cross-modal learning. In thispaper, we creatively model video-text as game players with multivariatecooperative game theory to wisely handle the uncertainty during fine-grainedsemantic interaction with diverse granularity, flexible combination, and vagueintensity. Concretely, we propose Hierarchical Banzhaf Interaction (HBI) tovalue possible correspondence between video frames and text words for sensitiveand explainable cross-modal contrast. To efficiently realize the cooperativegame of multiple video frames and multiple text words, the proposed methodclusters the original video frames (text words) and computes the BanzhafInteraction between the merged tokens. By stacking token merge modules, weachieve cooperative games at different semantic levels. Extensive experimentson commonly used text-video retrieval and video-question answering benchmarkswith superior performances justify the efficacy of our HBI. More encouragingly,it can also serve as a visualization tool to promote the understanding ofcross-modal interaction, which have a far-reaching impact on the community.Project page is available at https://jpthu17.github.io/HBI/.

Video-Text as Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learning