English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Existing works on weakly-supervised audio-visual video parsing adopt hybridattention network (HAN) as the multi-modal embedding to capture the cross-modalcontext. It embeds the audio and visual modalities with a shared network, wherethe cross-attention is performed at the input. However, such an early fusionmethod highly entangles the two non-fully correlated modalities and leads tosub-optimal performance in detecting single-modality events. To deal with thisproblem, we propose the messenger-guided mid-fusion transformer to reduce theuncorrelated cross-modal context in the fusion. The messengers condense thefull cross-modal context into a compact representation to only preserve usefulcross-modal information. Furthermore, due to the fact that microphones captureaudio events from all directions, while cameras only record visual eventswithin a restricted field of view, there is a more frequent occurrence ofunaligned cross-modal context from audio for visual event predictions. We thuspropose cross-audio prediction consistency to suppress the impact of irrelevantaudio information on visual event prediction. Experiments consistentlyillustrate the superior performance of our framework compared to existingstate-of-the-art methods.

Rethink Cross-Modal Fusion in Weakly-Supervised Audio-Visual Video Parsing