English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Audio-visual approaches involving visual inputs have laid the foundation forrecent progress in speech separation. However, the optimization of theconcurrent usage of auditory and visual inputs is still an active researcharea. Inspired by the cortico-thalamo-cortical circuit, in which the sensoryprocessing mechanisms of different modalities modulate one another via thenon-lemniscal sensory thalamus, we propose a novel cortico-thalamo-corticalneural network (CTCNet) for audio-visual speech separation (AVSS). First, theCTCNet learns hierarchical auditory and visual representations in a bottom-upmanner in separate auditory and visual subnetworks, mimicking the functions ofthe auditory and visual cortical areas. Then, inspired by the large number ofconnections between cortical regions and the thalamus, the model fuses theauditory and visual information in a thalamic subnetwork through top-downconnections. Finally, the model transmits this fused information back to theauditory and visual subnetworks, and the above process is repeated severaltimes. The results of experiments on three speech separation benchmark datasetsshow that CTCNet remarkably outperforms existing AVSS methods withconsiderablely fewer parameters. These results suggest that mimicking theanatomical connectome of the mammalian brain has great potential for advancingthe development of deep neural networks. Project repo ishttps://github.com/JusperLee/CTCNet.

An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuits