English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we address the problem of separating individual speech signalsfrom videos using audio-visual neural processing. Most conventional approachesutilize frame-wise matching criteria to extract shared information betweenco-occurring audio and video. Thus, their performance heavily depends on theaccuracy of audio-visual synchronization and the effectiveness of theirrepresentations. To overcome the frame discontinuity problem between twomodalities due to transmission delay mismatch or jitter, we propose across-modal affinity network (CaffNet) that learns global correspondence aswell as locally-varying affinities between audio and visual streams. Given thatthe global term provides stability over a temporal sequence at theutterance-level, this resolves the label permutation problem characterized byinconsistent assignments. By extending the proposed cross-modal affinity on thecomplex network, we further improve the separation performance in the complexspectral domain. Experimental results verify that the proposed methodsoutperform conventional ones on various datasets, demonstrating theiradvantages in real-world scenarios.

Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separation