English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Audio-visual speech recognition (AVSR) provides a promising solution toameliorate the noise-robustness of audio-only speech recognition with visualinformation. However, most existing efforts still focus on audio modality toimprove robustness considering its dominance in AVSR task, with noiseadaptation techniques such as front-end denoise processing. Though effective,these methods are usually faced with two practical challenges: 1) lack ofsufficient labeled noisy audio-visual training data in some real-worldscenarios and 2) less optimal model generality to unseen testing noises. Inthis work, we investigate the noise-invariant visual modality to strengthenrobustness of AVSR, which can adapt to any testing noises while withoutdependence on noisy training data, a.k.a., unsupervised noise adaptation.Inspired by human perception mechanism, we propose a universal viseme-phonememapping (UniVPM) approach to implement modality transfer, which can restoreclean audio from visual signals to enable speech recognition under any noisyconditions. Extensive experiments on public benchmarks LRS3 and LRS2 show thatour approach achieves the state-of-the-art under various noisy as well as cleanconditions. In addition, we also outperform previous state-of-the-arts onvisual speech recognition task.

Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition