English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently, large-scale Vision and Language (V\&L) pretraining has become thestandard backbone of many multimedia systems. While it has shown remarkableperformance even in unseen situations, it often performs in ways not intuitiveto humans. Particularly, they usually do not consider the pronunciation of theinput, which humans would utilize to understand language, especially when itcomes to unknown words. Thus, this paper inserts phonetic prior intoContrastive Language-Image Pretraining (CLIP), one of the V\&L pretrainedmodels, to make it consider the pronunciation similarity among itspronunciation inputs. To achieve this, we first propose a phoneme embeddingthat utilizes the phoneme relationships provided by the International PhoneticAlphabet (IPA) chart as a phonetic prior. Next, by distilling the frozen CLIPtext encoder, we train a pronunciation encoder employing the IPA-basedembedding. The proposed model named IPA-CLIP comprises this pronunciationencoder and the original CLIP encoders (image and text). Quantitativeevaluation reveals that the phoneme distribution on the embedding spacerepresents phonetic relationships more accurately when using the proposedphoneme embedding. Furthermore, in some multimodal retrieval tasks, we confirmthat the proposed pronunciation encoder enhances the performance of the textencoder and that the pronunciation encoder handles nonsense words in a morephonetic manner than the text encoder. Finally, qualitative evaluation verifiesthe correlation between the pronunciation encoder and human perceptionregarding pronunciation similarity.

IPA-CLIP: Integrating Phonetic Priors into Vision and Language Pretraining