English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Synthesizing realistic videos according to a given speech is still an openchallenge. Previous works have been plagued by issues such as inaccurate lipshape generation and poor image quality. The key reason is that only motionsand appearances on limited facial areas (e.g., lip area) are mainly driven bythe input speech. Therefore, directly learning a mapping function from speechto the entire head image is prone to ambiguity, particularly when using a shortvideo for training. We thus propose a decomposition-synthesis-compositionframework named Speech to Lip (Speech2Lip) that disentangles speech-sensitiveand speech-insensitive motion/appearance to facilitate effective learning fromlimited training data, resulting in the generation of natural-looking videos.First, given a fixed head pose (i.e., canonical space), we present aspeech-driven implicit model for lip image generation which concentrates onlearning speech-sensitive motion and appearance. Next, to model the majorspeech-insensitive motion (i.e., head movement), we introduce a geometry-awaremutual explicit mapping (GAMEM) module that establishes geometric mappingsbetween different head poses. This allows us to paste generated lip images atthe canonical space onto head images with arbitrary poses and synthesizetalking videos with natural head movements. In addition, a Blend-Net and acontrastive sync loss are introduced to enhance the overall synthesisperformance. Quantitative and qualitative results on three benchmarksdemonstrate that our model can be trained by a video of just a few minutes inlength and achieve state-of-the-art performance in both visual quality andspeech-visual synchronization. Code: https://github.com/CVMI-Lab/Speech2Lip.

Speech2Lip: High-fidelity Speech to Lip Generation by Learning from a Short Video