English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Despite the rapid advance of automatic speech recognition (ASR) technologies,accurate recognition of cocktail party speech characterised by the interferencefrom overlapping speakers, background noise and room reverberation remains ahighly challenging task to date. Motivated by the invariance of visual modalityto acoustic signal corruption, audio-visual speech enhancement techniques havebeen developed, although predominantly targeting overlapping speech separationand recognition tasks. In this paper, an audio-visual multi-channel speechseparation, dereverberation and recognition approach featuring a fullincorporation of visual information into all three stages of the system isproposed. The advantage of the additional visual modality over using audio onlyis demonstrated on two neural dereverberation approaches based on DNN-WPE andspectral mapping respectively. The learning cost function mismatch between theseparation and dereverberation models and their integration with the back-endrecognition system is minimised using fine-tuning on the MSE and LF-MMIcriteria. Experiments conducted on the LRS2 dataset suggest that the proposedaudio-visual multi-channel speech separation, dereverberation and recognitionsystem outperforms the baseline audio-visual multi-channel speech separationand recognition system containing no dereverberation module by a statisticallysignificant word error rate (WER) reduction of 2.06% absolute (8.77% relative).

Audio-visual multi-channel speech separation, dereverberation and recognition