English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Unsupervised pre-training is now the predominant approach for both text andspeech understanding. Self-attention models pre-trained on large amounts ofunannotated data have been hugely successful when fine-tuned on downstreamtasks from a variety of domains and languages. This paper takes theuniversality of unsupervised language pre-training one step further, byunifying speech and text pre-training within a single model. We build a singleencoder with the BERT objective on unlabeled text together with the w2v-BERTobjective on unlabeled speech. To further align our model representationsacross modalities, we leverage alignment losses, specifically TranslationLanguage Modeling (TLM) and Speech Text Matching (STM) that make use ofsupervised speech-text recognition data. We demonstrate that incorporating bothspeech and text data during pre-training can significantly improve downstreamquality on CoVoST~2 speech translation, by around 1 BLEU compared tosingle-modality pre-trained models, while retaining close to SotA performanceon LibriSpeech and SpeechStew ASR tasks. On four GLUE tasks andtext-normalization, we observe evidence of capacity limitations andinterference between the two modalities, leading to degraded performancecompared to an equivalent text-only model, while still being competitive withBERT. Through extensive empirical analysis we also demonstrate the importanceof the choice of objective function for speech pre-training, and the beneficialeffect of adding additional supervised signals on the quality of the learnedrepresentations.

SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training