English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

This paper explores sentence-level Multilingual Visual Speech Recognitionwith a single model for the first time. As the massive multilingual modeling ofvisual data requires huge computational costs, we propose a novel strategy,processing with visual speech units. Motivated by the recent success of theaudio speech unit, the proposed visual speech unit is obtained by discretizingthe visual speech features extracted from the self-supervised visual speechmodel. To correctly capture multilingual visual speech, we first train theself-supervised visual speech model on 5,512 hours of multilingual audio-visualdata. Through analysis, we verify that the visual speech units mainly containviseme information while suppressing non-linguistic information. By using thevisual speech units as the inputs of our system, we pre-train the model topredict corresponding text outputs on massive multilingual data constructed bymerging several VSR databases. As both the inputs and outputs are discrete, wecan greatly improve the training efficiency compared to the standard VSRtraining. Specifically, the input data size is reduced to 0.016% of theoriginal video inputs. In order to complement the insufficient visualinformation in speech recognition, we apply curriculum learning where theinputs of the system begin with audio-visual speech units and gradually changeto visual speech units. After pre-training, the model is finetuned oncontinuous features. We set new state-of-the-art multilingual VSR performancesby achieving comparable performances to the previous language-specific VSRmodels, with a single trained model.

Multilingual Visual Speech Recognition with a Single Model by Learning with Discrete Visual Speech Units