English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

While audio-visual speech models can yield superior performance androbustness compared to audio-only models, their development and adoption arehindered by the lack of labeled and unlabeled audio-visual data and the cost todeploy one model per modality. In this paper, we present u-HuBERT, aself-supervised pre-training framework that can leverage both multimodal andunimodal speech with a unified masked cluster prediction objective. Byutilizing modality dropout during pre-training, we demonstrate that a singlefine-tuned model can achieve performance on par or better than thestate-of-the-art modality-specific models. Moreover, our model fine-tuned onlyon audio can perform well with audio-visual and visual speech input, achievingzero-shot modality generalization for multiple speech processing tasks. Inparticular, our single model yields 1.2%/1.4%/27.2% speech recognition worderror rate on LRS3 with audio-visual/audio/visual input. Codes and models areavailable at https://github.com/facebookresearch/av_hubert

u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality