English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

In this paper, we introduce a new task, spoken video grounding (SVG), whichaims to localize the desired video fragments from spoken language descriptions.Compared with using text, employing audio requires the model to directlyexploit the useful phonemes and syllables related to the video from raw speech.Moreover, we randomly add environmental noises to this speech audio, furtherincreasing the difficulty of this task and better simulating real applications.To rectify the discriminative phonemes and extract video-related informationfrom noisy audio, we develop a novel video-guided curriculum learning (VGCL)during the audio pre-training process, which can make use of the vital visualperceptions to help understand the spoken language and suppress the externalnoise. Considering during inference the model can not obtain ground truth videosegments, we design a curriculum strategy that gradually shifts the input videofrom the ground truth to the entire video content during pre-training. Finally,the model can learn how to extract critical visual information from the entirevideo clip to help understand the spoken language. In addition, we collect thefirst large-scale spoken video grounding dataset based on ActivityNet, which isnamed as ActivityNet Speech dataset. Extensive experiments demonstrate ourproposed video-guided curriculum learning can facilitate the pre-trainingprocess to obtain a mutual audio encoder, significantly promoting theperformance of spoken video grounding tasks. Moreover, we prove that in thecase of noisy sound, our model outperforms the method that grounding video withASR transcripts, further demonstrating the effectiveness of our curriculumstrategy.

Video-Guided Curriculum Learning for Spoken Video Grounding