English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Video Object Grounding (VOG) is the problem of associating spatial objectregions in the video to a descriptive natural language query. This is achallenging vision-language task that necessitates constructing the correctcross-modal correspondence and modeling the appropriate spatio-temporal contextof the query video and caption, thereby localizing the specific objectsaccurately. In this paper, we tackle this task by a novel framework calledHiErarchical spatio-tempoRal reasOning (HERO) with contrastive actioncorrespondence. We study the VOG task at two aspects that prior worksoverlooked: (1) Contrastive Action Correspondence-aware Retrieval. Notice thatthe fine-grained video semantics (e.g., multiple actions) is not totallyaligned with the annotated language query (e.g., single action), we firstintroduce the weakly-supervised contrastive learning that classifies the videoas action-consistent and action-independent frames relying on the video-captionaction semantic correspondence. Such a design can build the fine-grainedcross-modal correspondence for more accurate subsequent VOG. (2) HierarchicalSpatio-temporal Modeling Improvement. While transformer-based VOG modelspresent their potential in sequential modality (i.e., video and caption)modeling, existing evidence also indicates that the transformer suffers fromthe issue of the insensitive spatio-temporal locality. Motivated by that, wecarefully design the hierarchical reasoning layers to decouple fully connectedmulti-head attention and remove the redundant interfering correlations.Furthermore, our proposed pyramid and shifted alignment mechanisms areeffective to improve the cross-modal information utilization of neighborhoodspatial regions and temporal frames. We conducted extensive experiments to showour HERO outperforms existing techniques by achieving significant improvementon two benchmark datasets.

HERO: HiErarchical spatio-tempoRal reasOning with Contrastive Action Correspondence for End-to-End Video Object Grounding