English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently reported state-of-the-art results in visual speech recognition (VSR)often rely on increasingly large amounts of video data, while the publiclyavailable transcribed video datasets are limited in size. In this paper, forthe first time, we study the potential of leveraging synthetic visual data forVSR. Our method, termed SynthVSR, substantially improves the performance of VSRsystems with synthetic lip movements. The key idea behind SynthVSR is toleverage a speech-driven lip animation model that generates lip movementsconditioned on the input speech. The speech-driven lip animation model istrained on an unlabeled audio-visual dataset and could be further optimizedtowards a pre-trained VSR model when labeled videos are available. As plenty oftranscribed acoustic data and face images are available, we are able togenerate large-scale synthetic data using the proposed lip animation model forsemi-supervised VSR training. We evaluate the performance of our approach onthe largest public VSR benchmark - Lip Reading Sentences 3 (LRS3). SynthVSRachieves a WER of 43.3% with only 30 hours of real labeled data, outperformingoff-the-shelf approaches using thousands of hours of video. The WER is furtherreduced to 27.9% when using all 438 hours of labeled data from LRS3, which ison par with the state-of-the-art self-supervised AV-HuBERT method. Furthermore,when combined with large-scale pseudo-labeled audio-visual data SynthVSR yieldsa new state-of-the-art VSR WER of 16.9% using publicly available data only,surpassing the recent state-of-the-art approaches trained with 29 times morenon-public machine-transcribed video data (90,000 hours). Finally, we performextensive ablation studies to understand the effect of each component in ourproposed method.

SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision