English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Lip-to-Speech (Lip2Speech) synthesis, which predicts corresponding speechfrom talking face images, has witnessed significant progress with variousmodels and training strategies in a series of independent studies. However,existing studies can not achieve voice control under zero-shot condition,because extra speaker embeddings need to be extracted from natural referencespeech and are unavailable when only the silent video of an unseen speaker isgiven. In this paper, we propose a zero-shot personalized Lip2Speech synthesismethod, in which face images control speaker identities. A variationalautoencoder is adopted to disentangle the speaker identity and linguisticcontent representations, which enables speaker embeddings to control the voicecharacteristics of synthetic speech for unseen speakers. Furthermore, wepropose associated cross-modal representation learning to promote the abilityof face-based speaker embeddings (FSE) on voice control. Extensive experimentsverify the effectiveness of the proposed method whose synthetic utterances aremore natural and matching with the personality of input video than the comparedmethods. To our best knowledge, this paper makes the first attempt on zero-shotpersonalized Lip2Speech synthesis with a face image rather than reference audioto control voice characteristics.

Zero-shot personalized lip-to-speech synthesis with face image based voice control