English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Audio-visual speech separation methods aim to integrate different modalitiesto generate high-quality separated speech, thereby enhancing the performance ofdownstream tasks such as speech recognition. Most existing state-of-the-art(SOTA) models operate in the time domain. However, their overly simplisticapproach to modeling acoustic features often necessitates larger and morecomputationally intensive models in order to achieve SOTA performance. In thispaper, we present a novel time-frequency domain audio-visual speech separationmethod: Recurrent Time-Frequency Separation Network (RTFS-Net), which appliesits algorithms on the complex time-frequency bins yielded by the Short-TimeFourier Transform. We model and capture the time and frequency dimensions ofthe audio independently using a multi-layered RNN along each dimension.Furthermore, we introduce a unique attention-based fusion technique for theefficient integration of audio and visual information, and a new maskseparation approach that takes advantage of the intrinsic spectral nature ofthe acoustic features for a clearer separation. RTFS-Net outperforms theprevious SOTA method using only 10% of the parameters and 18% of the MACs. Thisis the first time-frequency domain audio-visual speech separation method tooutperform all contemporary time-domain counterparts.

RTFS-Net: Recurrent time-frequency modelling for efficient audio-visual speech separation