English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent work has studied text-to-audio synthesis using large amounts of pairedtext-audio data. However, audio recordings with high-quality text annotationscan be difficult to acquire. In this work, we approach text-to-audio synthesisusing unlabeled videos and pretrained language-vision models. We propose tolearn the desired text-audio correspondence by leveraging the visual modalityas a bridge. We train a conditional diffusion model to generate the audio trackof a video, given a video frame encoded by a pretrained contrastivelanguage-image pretraining (CLIP) model. At test time, we first exploreperforming a zero-shot modality transfer and condition the diffusion model witha CLIP-encoded text query. However, we observe a noticeable performance dropwith respect to image queries. To close this gap, we further adopt a pretraineddiffusion prior model to generate a CLIP image embedding given a CLIP textembedding. Our results show the effectiveness of the proposed method, and thatthe pretrained diffusion prior can reduce the modality transfer gap. While wefocus on text-to-audio synthesis, the proposed model can also generate audiofrom image queries, and it shows competitive performance against astate-of-the-art image-to-audio synthesis model in a subjective listening test.This study offers a new direction of approaching text-to-audio synthesis thatleverages the naturally-occurring audio-visual correspondence in videos and thepower of pretrained language-vision models.

CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained Language-Vision Models