English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Motivated by the success of masked language modeling~(MLM) in pre-trainingnatural language processing models, we propose w2v-BERT that explores MLM forself-supervised speech representation learning. w2v-BERT is a framework thatcombines contrastive learning and MLM, where the former trains the model todiscretize input continuous speech signals into a finite set of discriminativespeech tokens, and the latter trains the model to learn contextualized speechrepresentations via solving a masked prediction task consuming the discretizedtokens. In contrast to existing MLM-based speech pre-training frameworks suchas HuBERT, which relies on an iterative re-clustering and re-training process,or vq-wav2vec, which concatenates two separately trained modules, w2v-BERT canbe optimized in an end-to-end fashion by solving the two self-supervisedtasks~(the contrastive task and MLM) simultaneously. Our experiments show thatw2v-BERT achieves competitive results compared to current state-of-the-artpre-trained models on the LibriSpeech benchmarks when using the Libri-Light~60kcorpus as the unsupervised data. In particular, when compared to publishedmodels such as conformer-based wav2vec~2.0 and HuBERT, our model shows~5\%to~10\% relative WER reduction on the test-clean and test-other subsets. Whenapplied to the Google's Voice Search traffic dataset, w2v-BERT outperforms ourinternal conformer-based wav2vec~2.0 by more than~30\% relatively.

W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training