English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Visual information can serve as an effective cue for target speakerextraction (TSE) and is vital to improving extraction performance. In thispaper, we propose AV-SepFormer, a SepFormer-based attention dual-scale modelthat utilizes cross- and self-attention to fuse and model features from audioand visual. AV-SepFormer splits the audio feature into a number of chunks,equivalent to the length of the visual feature. Then self- and cross-attentionare employed to model and fuse the multi-modal features. Furthermore, we use anovel 2D positional encoding, that introduces the positional informationbetween and within chunks and provides significant gains over the traditionalpositional encoding. Our model has two key advantages: the time granularity ofaudio chunked feature is synchronized to the visual feature, which alleviatesthe harm caused by the inconsistency of audio and video sampling rate; bycombining self- and cross-attention, feature fusion and speech extractionprocesses are unified within an attention paradigm. The experimental resultsshow that AV-SepFormer significantly outperforms other existing methods.

AV-SepFormer: Cross-Attention SepFormer for Audio-Visual Target Speaker Extraction