English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently, video object segmentation (VOS) referred by multi-modal signals,e.g., language and audio, has evoked increasing attention in both industry andacademia. It is challenging for exploring the semantic alignment withinmodalities and the visual correspondence across frames. However, existingmethods adopt separate network architectures for different modalities, andneglect the inter-frame temporal interaction with references. In this paper, wepropose MUTR, a Multi-modal Unified Temporal transformer for Referring videoobject segmentation. With a unified framework for the first time, MUTR adopts aDETR-style transformer and is capable of segmenting video objects designated byeither text or audio reference. Specifically, we introduce two strategies tofully explore the temporal relations between videos and multi-modal signals.Firstly, for low-level temporal aggregation before the transformer, we enablethe multi-modal references to capture multi-scale visual cues from consecutivevideo frames. This effectively endows the text or audio signals with temporalknowledge and boosts the semantic alignment between modalities. Secondly, forhigh-level temporal interaction after the transformer, we conduct inter-framefeature communication for different object embeddings, contributing to betterobject-wise correspondence for tracking along the video. On Ref-YouTube-VOS andAVSBench datasets with respective text and audio references, MUTR achieves+4.2% and +8.7% J&F improvements to state-of-the-art methods, demonstrating oursignificance for unified multi-modal VOS. Code is released athttps://github.com/OpenGVLab/MUTR.

Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation