English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

We propose the (<inline-formula><tex-math notation="LaTeX">$\tt SAMU\text{-}XLSR$</tex-math></inline-formula>): <underline>S</underline>emantically-<underline>A</underline>ligned <underline>M</underline>ultimodal <underline>U</underline>tterance-level <underline>Cross</underline>-<underline>L</underline>ingual <underline>S</underline>peech <underline>R</underline>epresentation learning framework. Unlike previous works on speech representation learning, which learns multilingual contextual speech embedding at the resolution of an acoustic frame (10&#x2013;20 ms), this work focuses on learning multimodal (speech-text) multilingual speech embedding at the resolution of a sentence (5&#x2013;10 s) such that the embedding vector space is semantically aligned across different languages. We combine state-of-the-art multilingual acoustic frame-level speech representation learning model <inline-formula><tex-math notation="LaTeX">$\tt XLSR$</tex-math></inline-formula> with the Language Agnostic BERT Sentence Embedding (<inline-formula><tex-math notation="LaTeX">$\tt LaBSE$</tex-math></inline-formula>) model to create an utterance-level multimodal multilingual speech encoder <inline-formula><tex-math notation="LaTeX">$\tt SAMU\text{-}XLSR$</tex-math></inline-formula>. Although we train <inline-formula><tex-math notation="LaTeX">$\tt SAMU\text{-}XLSR$</tex-math></inline-formula> with only multilingual transcribed speech data, cross-lingual speech-text and speech-speech associations emerge in its learned representation space. To substantiate our claims, we use <inline-formula><tex-math notation="LaTeX">$\tt SAMU\text{-}XLSR$</tex-math></inline-formula> speech encoder in combination with a pre-trained <inline-formula><tex-math notation="LaTeX">$\tt LaBSE$</tex-math></inline-formula> text sentence encoder for cross-lingual speech-to-text translation retrieval, and <inline-formula><tex-math notation="LaTeX">$\tt SAMU\text{-}XLSR$</tex-math></inline-formula> alone for cross-lingual speech-to-speech translation retrieval. We highlight these applications by performing several cross-lingual text and speech translation retrieval tasks across several datasets.

SAMU-XLSR: Semantically-Aligned Multimodal Utterance-Level Cross-Lingual Speech Representation