English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

While recent research has made significant progress in speech-driven talkingface generation, the quality of the generated video still lags behind that ofreal recordings. One reason for this is the use of handcrafted intermediaterepresentations like facial landmarks and 3DMM coefficients, which are designedbased on human knowledge and are insufficient to precisely describe facialmovements. Additionally, these methods require an external pretrained model forextracting these representations, whose performance sets an upper bound ontalking face generation. To address these limitations, we propose a novelmethod called DAE-Talker that leverages data-driven latent representationsobtained from a diffusion autoencoder (DAE). DAE contains an image encoder thatencodes an image into a latent vector and a DDIM image decoder thatreconstructs the image from it. We train our DAE on talking face video framesand then extract their latent representations as the training target for aConformer-based speech2latent model. This allows DAE-Talker to synthesize fullvideo frames and produce natural head movements that align with the content ofspeech, rather than relying on a predetermined head pose from a template video.We also introduce pose modelling in speech2latent for pose controllability.Additionally, we propose a novel method for generating continuous video frameswith the DDIM image decoder trained on individual frames, eliminating the needfor modelling the joint distribution of consecutive frames directly. Ourexperiments show that DAE-Talker outperforms existing popular methods inlip-sync, video fidelity, and pose naturalness. We also conduct ablationstudies to analyze the effectiveness of the proposed techniques and demonstratethe pose controllability of DAE-Talker.

DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder