English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Finding the right sound effects (SFX) to match moments in a video is adifficult and time-consuming task, and relies heavily on the quality andcompleteness of text metadata. Retrieving high-quality (HQ) SFX using a videoframe directly as the query is an attractive alternative, removing the relianceon text metadata and providing a low barrier to entry for non-experts. Due tothe lack of HQ audio-visual training data, previous work on audio-visualretrieval relies on YouTube (in-the-wild) videos of varied quality fortraining, where the audio is often noisy and the video of amateur quality. Assuch it is unclear whether these systems would generalize to the task ofmatching HQ audio to production-quality video. To address this, we propose amultimodal framework for recommending HQ SFX given a video frame by (1)leveraging large language models and foundational vision-language models tobridge HQ audio and video to create audio-visual pairs, resulting in a highlyscalable automatic audio-visual data curation pipeline; and (2) usingpre-trained audio and visual encoders to train a contrastive learning-basedretrieval system. We show that our system, trained using our automatic datacuration pipeline, significantly outperforms baselines trained on in-the-wilddata on the task of HQ SFX retrieval for video. Furthermore, while thebaselines fail to generalize to this task, our system generalizes well fromclean to in-the-wild data, outperforming the baselines on a dataset of YouTubevideos despite only being trained on the HQ audio-visual pairs. A user studyconfirms that people prefer SFX retrieved by our system over the baseline 67%of the time both for HQ and in-the-wild data. Finally, we present ablations todetermine the impact of model and data pipeline design choices on downstreamretrieval performance. Please visit our project website to listen to and viewour SFX retrieval results.

Bridging High-Quality Audio and Video via Language for Sound Effects Retrieval from Visual Queries