English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We propose the first joint audio-video generation framework that bringsengaging watching and listening experiences simultaneously, towardshigh-quality realistic videos. To generate joint audio-video pairs, we proposea novel Multi-Modal Diffusion model (i.e., MM-Diffusion), with two-coupleddenoising autoencoders. In contrast to existing single-modal diffusion models,MM-Diffusion consists of a sequential multi-modal U-Net for a joint denoisingprocess by design. Two subnets for audio and video learn to gradually generatealigned audio-video pairs from Gaussian noises. To ensure semantic consistencyacross modalities, we propose a novel random-shift based attention blockbridging over the two subnets, which enables efficient cross-modal alignment,and thus reinforces the audio-video fidelity for each other. Extensiveexperiments show superior results in unconditional audio-video generation, andzero-shot conditional tasks (e.g., video-to-audio). In particular, we achievethe best FVD and FAD on Landscape and AIST++ dancing datasets. Turing tests of10k votes further demonstrate dominant preferences for our model. The code andpre-trained models can be downloaded athttps://github.com/researchmm/MM-Diffusion.

MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation