English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The challenge of talking face generation from speech lies in aligning twodifferent modal information, audio and video, such that the mouth regioncorresponds to input audio. Previous methods either exploit audio-visualrepresentation learning or leverage intermediate structural information such aslandmarks and 3D models. However, they struggle to synthesize fine details ofthe lips varying at the phoneme level as they do not sufficiently providevisual information of the lips at the video synthesis step. To overcome thislimitation, our work proposes Audio-Lip Memory that brings in visualinformation of the mouth region corresponding to input audio and enforcesfine-grained audio-visual coherence. It stores lip motion features fromsequential ground truth images in the value memory and aligns them withcorresponding audio features so that they can be retrieved using audio input atinference time. Therefore, using the retrieved lip motion features as visualhints, it can easily correlate audio with visual dynamics in the synthesisstep. By analyzing the memory, we demonstrate that unique lip features arestored in each memory slot at the phoneme level, capturing subtle lip motionbased on memory addressing. In addition, we introduce visual-visualsynchronization loss which can enhance lip-syncing performance when used alongwith audio-visual synchronization loss in our model. Extensive experiments areperformed to verify that our method generates high-quality video with mouthshapes that best align with the input audio, outperforming previousstate-of-the-art methods.

SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memory