English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We propose an Explicit Conditional Multimodal Variational Auto-Encoder(ECMVAE) for audio-visual segmentation (AVS), aiming to segment sound sourcesin the video sequence. Existing AVS methods focus on implicit feature fusionstrategies, where models are trained to fit the discrete samples in thedataset. With a limited and less diverse dataset, the resulting performance isusually unsatisfactory. In contrast, we address this problem from an effectiverepresentation learning perspective, aiming to model the contribution of eachmodality explicitly. Specifically, we find that audio contains criticalcategory information of the sound producers, and visual data provides candidatesound producer(s). Their shared information corresponds to the target soundproducer(s) shown in the visual data. In this case, cross-modal sharedrepresentation learning is especially important for AVS. To achieve this, ourECMVAE factorizes the representations of each modality with a modality-sharedrepresentation and a modality-specific representation. An orthogonalityconstraint is applied between the shared and specific representations tomaintain the exclusive attribute of the factorized latent code. Further, amutual information maximization regularizer is introduced to achieve extensiveexploration of each modality. Quantitative and qualitative evaluations on theAVSBench demonstrate the effectiveness of our approach, leading to a newstate-of-the-art for AVS, with a 3.84 mIOU performance leap on the challengingMS3 subset for multiple sound source segmentation.

Multimodal Variational Auto-encoder based Audio-Visual Segmentation