English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Given a piece of text, a video clip and a reference audio, the movie dubbing(also known as visual voice clone V2C) task aims to generate speeches thatmatch the speaker's emotion presented in the video using the desired speakervoice as reference. V2C is more challenging than conventional text-to-speechtasks as it additionally requires the generated speech to exactly match thevarying emotions and speaking speed presented in the video. Unlike previousworks, we propose a novel movie dubbing architecture to tackle these problemsvia hierarchical prosody modelling, which bridges the visual information tocorresponding speech prosody from three aspects: lip, face, and scene.Specifically, we align lip movement to the speech duration, and convey facialexpression to speech energy and pitch via attention mechanism based on valenceand arousal representations inspired by recent psychology findings. Moreover,we design an emotion booster to capture the atmosphere from global videoscenes. All these embeddings together are used to generate mel-spectrogram andthen convert to speech waves via existing vocoder. Extensive experimentalresults on the Chem and V2C benchmark datasets demonstrate the favorableperformance of the proposed method. The source code and trained models will bereleased to the public.

Learning to Dub Movies via Hierarchical Prosody Models