English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Lip-to-speech involves generating a natural-sounding speech synchronized witha soundless video of a person talking. Despite recent advances, current methodsstill cannot produce high-quality speech with high levels of intelligibilityfor challenging and realistic datasets such as LRS3. In this work, we presentLipVoicer, a novel method that generates high-quality speech, even forin-the-wild and rich datasets, by incorporating the text modality. Given asilent video, we first predict the spoken text using a pre-trained lip-readingnetwork. We then condition a diffusion model on the video and use the extractedtext through a classifier-guidance mechanism where a pre-trained ASR serves asthe classifier. LipVoicer outperforms multiple lip-to-speech baselines on LRS2and LRS3, which are in-the-wild datasets with hundreds of unique speakers intheir test set and an unrestricted vocabulary. Moreover, our experiments showthat the inclusion of the text modality plays a major role in theintelligibility of the produced speech, readily perceptible while listening,and is empirically reflected in the substantial reduction of the WER metric. Wedemonstrate the effectiveness of LipVoicer through human evaluation, whichshows that it produces more natural and synchronized speech signals compared tocompeting methods. Finally, we created a demo showcasing LipVoicer'ssuperiority in producing natural, synchronized, and intelligible speech,providing additional evidence of its effectiveness. Project page:https://lipvoicer.github.io

LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading