English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In recent research, slight performance improvement is observed from automaticspeech recognition systems to audio-visual speech recognition systems in theend-to-end framework with low-quality videos. Unmatching convergence rates andspecialized input representations between audio and visual modalities areconsidered to cause the problem. In this paper, we propose two novel techniquesto improve audio-visual speech recognition (AVSR) under a pre-training andfine-tuning training framework. First, we explore the correlation between lipshapes and syllable-level subword units in Mandarin to establish goodframe-level syllable boundaries from lip shapes. This enables accuratealignment of video and audio streams during visual model pre-training andcross-modal fusion. Next, we propose an audio-guided cross-modal fusion encoder(CMFE) neural network to utilize main training parameters for multiplecross-modal attention layers to make full use of modality complementarity.Experiments on the MISP2021-AVSR data set show the effectiveness of the twoproposed techniques. Together, using only a relatively small amount of trainingdata, the final system achieves better performances than state-of-the-artsystems with more complex front-ends and back-ends.

Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder