English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Visually grounded speech models learn from images paired with spokencaptions. By tagging images with soft text labels using a trained visualclassifier with a fixed vocabulary, previous work has shown that it is possibleto train a model that can detect whether a particular text keyword occurs inspeech utterances or not. Here we investigate whether visually grounded speechmodels can also do keyword localisation: predicting where, within an utterance,a given textual keyword occurs without any explicit text-based or alignmentsupervision. We specifically consider whether incorporating attention into aconvolutional model is beneficial for localisation. Although absolutelocalisation performance with visually supervised models is still modest(compared to using unordered bag-of-word text labels for supervision), we showthat attention provides a large gain in performance over previous visuallygrounded models. As in many other speech-image studies, we find that many ofthe incorrect localisations are due to semantic confusions, e.g. locating theword 'backstroke' for the query keyword 'swimming'.

Attention-Based Keyword Localisation in Speech using Visual Grounding