English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The objective of this paper is to perform audio-visual sound sourceseparation, i.e.~to separate component audios from a mixture based on thevideos of sound sources. Moreover, we aim to pinpoint the source location inthe input video sequence. Recent works have shown impressive audio-visualseparation results when using prior knowledge of the source type (e.g. humanplaying instrument) and pre-trained motion detectors (e.g. keypoints or opticalflows). However, at the same time, the models are limited to a certainapplication domain. In this paper, we address these limitations and make thefollowing contributions: i) we propose a two-stage architecture, calledAppearance and Motion network (AMnet), where the stages specialise toappearance and motion cues, respectively. The entire system is trained in aself-supervised manner; ii) we introduce an Audio-Motion Embedding (AME)framework to explicitly represent the motions that related to sound; iii) wepropose an audio-motion transformer architecture for audio and motion featurefusion; iv) we demonstrate state-of-the-art performance on two challengingdatasets (MUSIC-21 and AVE) despite the fact that we do not use any pre-trainedkeypoint detectors or optical flow estimators. Project page:https://ly-zhu.github.io/self-supervised-motion-representations

Visually Guided Sound Source Separation and Localization using Self-Supervised Motion Representations